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Preface 


For  the  twenty-seventh  year,  the  Research  and  Theory  Division  of  the  Association  for  Educational 
Communications  and  Technology  (AECT)  is  sponsoring  the  publication  of  these  Proceedings.  Papers 
published  in  this  volume  were  presented  at  the  National  AECT  Convention  in  Chicago,  IL.  A limited 
quantity  of  these  Proceedings  were  printed  and  sold  in  both  hardcopy  and  electronic  versions. copies  of  both 
volumes  were  distributed  to  Convention  attendees  on  compact  disk.  Volume  #1  will  be  available  on 
microfiche  through  the  Educational  Resources  Clearinghouse  (ERIC)  System. 

The  Proceedings  of  AECT’s  Convention  are  published  in  two  volumes.  Volume  #1  contains  papers  dealing 
primarily  with  research  and  development  topics.  Papers  dealing  with  instruction  and  training  issues  are 
contained  in  volume  #2  which  also  contains  over  100  papers. 

REFEREEING  PROCESS:  Papers  selected  for  presentation  at  the  AECT  Convention  and  included  in  these 
Proceedings  were  subjected  to  a reviewing  process.  All  references  to  authorship  were  removed  from 
proposals  before  they  were  submitted  to  referees  for  review.  Approximately  sixty  percent  of  the 
manuscripts  submitted  for  consideration  were  selected  for  presentation  at  the  convention  and  for 
publication  in  these  Proceedings.  The  papers  contained  in  this  document  represent  some  of  the  most  current 
thinking  in  educational  communications  and  technology. 
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Improving  School  Children’s  Mathematical  Word  Problem  Solving  Skills 
through  Computer-Based  Multiple  Representations 

Tufan  Adiguzel 
Texas  A&M  University 

Yavuz  Akpinar 
Bogazici  University 

Abstract 

Instructional  resources  that  employ  multiple  representations  have  become  commonplace  in 
mathematics  classrooms.  This  study  will  present  computer  software,  LaborScale  which  was  designed  to  improve 
seventh  grade  students’  word  problem-solving  skills  through  computer-based  multiple  representations  including 
graphic,  symbolic,  and  audio  representations.  The  proposed  presentation  will  illustrate  the  design, 
implementation  and  validation  of  an  interactive  learning  environment  (ILE),  LaborScale.  This  1LE  is  based 
upon  the  principles  of  computer  based  interactive  problem  solving  environments,  which  connects  different  types 
of  knowledge  representation  forms,  and  aims  primarily  to  assist  students  as  they  explore  symbolic 
representations  used  in  word-problem  solving  process. 

Problem  Solving 

When  solving  problems,  a learner  combines  previously  learned  elements  of  knowledge,  rules, 
techniques,  skills , and  concepts  to  provide  a solution  to  a novel  situation.  It  is  generally  accepted  that 
mathematics  is  both  process  and  product:  both  an  organized  body  of  knowledge  and  a creative  activity  in  which 
the  learner  participates . It  might,  in  fact,  be  claimed  that  the  real  purpose  of  learning  rules,  techniques,  and 
content  is  to  enable  the  learner  to  do  mathematics,  indeed  to  solve  problems  (Orton,  1987).  Thus  problem 
solving  can  be  considered  to  be  the  real  essence  of  mathematics.  Gagne  (1985)  has  expressed  the  view  that 
problem  solving  is  the  highest  form  of  learning.  Having  solved  a problem,  one  has  learned.  One  might  only 
have  learned  to  solve  that  problem,  but  it  is  more  likely  that  one  has  learned  to  solve  a variety  of  similar 
problems  and  perhaps  even  a variety  of  problems  possessing  some  similar  characteristics.  Jonassen,  Howland, 
Moore,  and  Marra  (2003)  also  point  out  that  solving  problems  are  meaningful  kind  of  learning  activity  in 
educational  settings. 

Problem  solving  activities  introduce  difficulties  for  management  by  teachers.  For  example:  choosing 
and  sequencing  problem  solving  tasks,  determining  the  degree  and  type  of  assistance  to  be  given  to  students, 
maintaining  motivation,  and  knowing  how  to  consolidate  understanding  through  reflection  and  follow-up 
discussions,  demand  continuous  decision  making  on  the  part  of  teachers.  Similarly,  learners  have  complicated 
tasks  to  complete  during  problem  solving,  especially  when  solving  word  problems.  Word  problems  are  set  in 
specific  contexts  from  which  students  have  to  develop  representations.  Learners  have  to  link  representations  to 
appropriate  mathematical  formulations,  and  to  apply  appropriate  techniques  in  order  to  produce  a solution. 
Learners  have  to  monitor  and  evaluate  this  problem  solving  process,  and  consider  the  implications  of  the 
solution.  In  consequence,  giving  assistance  and  managing  problem  solving  in  ways  which  create  effective 
learning  and  the  development  of  cognitive  skills  is  not  an  easy  task. 

Research  shows  that  a major  source  of  difficulty  experienced  by  children  in  the  problem  solving 
process  is  transforming  the  written  word  into  mathematical  operations  and  the  symbolization  of  these 
operations.  Namely,  children  are  required  to  disembed  the  information  from  the  problem  context,  select  the 
relevant  values,  and  insert  them  into  some  formula.  However,  children  are  not  very  successful  in  transferring 
their  abilities  to  solve  problems  to  subsequent  problems  (Jonassen  et  ah,  2003).  Orton  (1987)  indicated  that  the 
most  common  difficulty  of  problem  solving  is  failure  to  use  known  information.  He  also  noted  that  in  order  to 
cope  with  this  difficulty  pupils  should  (1)  write  the  problem  in  primitive  form  and  sketch  an  accurate  picture  of 
the  setup  (where  applicable),  (2)  transform  the  primitive  statements  to  simpler  language,  and  (3)  translate  verbal 
problems  to  more  abstract  mathematical  statement(s)  and  figures,  diagrams,  charts  and  other  similar 
representations. 
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Multiple  Representations 

Recent  approaches  to  mathematics  instruction  in  the  classroom  emphasize  mathematics  as  flexible, 
insightful  problem  solving  that  requires  understanding  that  mathematics  involves  pattern  seeking, 
experimentation,  hypothesis  testing,  and  active  seeking  of  solutions.  But  children’s  beliefs  about  the  nature  of 
mathematics  contrast  with  this  emphasis . For  example,  Baroody  (1987)  asserts  that  due  to  an  overemphasis  on 
‘the  right  answer’,  children  commonly  believe  that  all  problems  must  have  a correct  answer,  that  there  is  only 
one  correct  way  to  solve  a problem  and  that  inexact  answers  or  procedures  (such  as  estimates)  are  undesirable. 
In  order  to  recognize  that  multiple  solutions  and  different  representations  of  problems  are  possible,  children 
need  to  have  higher  order  problem  solving  skills  . Polya  (1962)  advocated  that  solvers  should  choose  multiple 
representations  when  they  begin  to  solve  a problem.  Jiang  and  Mcclintock  (2000)  also  suggested  that 
encouraging  multiple  solutions  to  problem  solving  plays  an  important  role  in  facilitating  students’ 
understanding  of  mathematical  concepts  and  their  grasp  of  methods  of  mathematical  thinking.  In  this  way,  the 
National  Council  of  Teachers  of  Mathematics  (NCTM,  2000)  states , “representations  should  be  treated  as 
essential  elements  in  supporting  students’  understanding  of  mathematical  concepts  and  relationships;  in 
communicating  mathematical  approaches,  arguments,  and  understandings  to  one’s  self  and  to  others;  in 
recognizing  connections  among  related  mathematical  concepts;  and  in  applying  mathematics  to  realistic 
problem  situations  through  modeling’’  (p.  67). 

Representations  are  mainly  divided  into  two  categories.  External  representations  are  the  knowledge 
and  structure  in  the  environment,  as  physical  symbols,  objects,  or  dimensions  and  as  external  rules,  constraints, 
or  relations  embedded  in  physical  configurations  (Zhang,  1997,  p.  180).  Internal  representations  are  retrieved 
from  memory  by  cognitive  processes . External  and  internal  representations  are  particularly  beneficial  for 
learning  when  they  are  multiple.  In  most  cases,  learners  have  to  process  multiple  representations,  including 
graphics,  symbols  and  audio.  Classroom  teaching  has  traditionally  employed  multiple  external  representations 
(MERs)  in  the  pursuit  of  helping  students  learn.  Teachers  use  MERs  explicitly  in  order  to  make  abstract 
situations  more  concrete. 

Kaput  (1992)  proposed  that  multiple  linked  representations  might  allow  learners  to  perceive  complex 
ideas  in  a new  way  and  to  apply  them  more  effectively.  By  providing  a rich  source  of  representations  of  a 
domain,  one  can  supply  learners  with  opportunities  to  build  references  across  these  representations.  Such 
knowledge  can  be  used  to  expose  underlying  structure  in  the  domain  represented.  According  to  this  view, 
mathematics  knowledge  can  be  characterized  as  the  ability  to  construct  and  map  across  different 
representations. 


Computer-Based  Multiple  Representations 

Computer  environments  have  been  gaining  great  importance  in  education.  Numerical  computation 
tools  can  be  used  by  problem  solvers  to  emphasize  planning  and  interpretation  of  arithmetic  operations.  The 
existence  of  computer  graphics  tools  can  be  used  to  help  students  understand  abstract  mathematical  concepts,  to 
create  entirely  new  graphic  oriented  representations  of  traditional  mathematical  topics,  or  to  provide  alternative 
visual  methods  in  mathematical  problem  solving.  As  Kaput  (1992)  states  for  the  case  of  mathematics  education, 
this  entails  that  routine  computations  can  be  off-loaded  to  a machine,  that  new  representational  mechanisms 
only  available  on  computers  (such  as  programs  as  representations)  become  available,  and  that  one  can  reify 
abstract  concepts  by  means  of  computer  simulations,  making  them  more  readily  accessible  for  reflection  and 
dialogue. 

Fey  (1989)  asserts  that  the  use  of  numerical,  graphic  and  symbol  manipulation  is  a powerful  technique 
for  mathematics  teaching  and  learning.  He  identified  several  ways  in  which  computer-based  representations  of 
mathematical  ideas  are  unique  and  especially  promising  as  instructional  and  problem  solving.  First,  computer 
representations  of  mathematical  ideas  and  procedures  can  be  made  dynamic  in  ways  that  no  text  or  chalkboard 
diagram  can.  Second,  the  computer  makes  it  possible  to  offer  individual  students  an  environment  for  work  with 
representations  that  are  flexible,  but  at  the  same  time,  constrained  to  give  corrective  feedback  to  each  individual 
user  whenever  appropriate.  Third,  the  electronic  representation  plays  a role  in  helping  move  students  from 
concrete  thinking  about  an  idea  or  procedure  to  an  ultimately  more  powerful  abstract  symbolic  form.  Fourth,  the 
versatility  of  computer  graphics  has  made  it  possible  to  give  entirely  new  kinds  of  representations  for 
mathematics -representation  that  can  be  created  by  each  computer  user  to  suit  particular  purposes.  Finally,  the 
machine  accuracy  of  computer  generated  numerical,  graphic,  and  symbolic  representations  make  those 
computer  representations  available  as  powerful  new  tools  for  actually  solving  problems  (p.  255). 
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Designing  the  Interactive  Learning 

The  use  of  multimedia  technology  has  offered  an  alternative  way  of  delivering  instruction.  The  old 
text -based  approach  to  learning  is  being  superseded  by  an  approach,  which  includes  multisensory 
representations  (Jonassen  et  al.,  2003).  Interactive  multimedia  is  one  of  the  most  promising  technologies  of  the 
time  and  has  the  potential  to  revolutionize  the  way  we  work,  learn,  and  communicate  (Macromedia,  1992;  Staub 
& Wertherbe,  1989).  Interactive  multimedia  programs  take  the  idea  of  learning  and  doing  seriously.  With 
interactive  multimedia  programs , the  learning  process  is  modified  by  the  actions  of  the  learners,  thus  changing 
the  roles  of  both  the  learner  and  the  teacher.  Interactive  multimedia  learning  is  also  a process,  rather  than  a 
technology,  that  places  new  learning  potential  into  the  hands  of  users  (Jonassen,  1999).  The  ideal  interactive 
learning  environment  (ILE),  then,  is  one  where  students  are  encouraged  to  undertake  such  activities  and  are 
provided  with  feedback  as  they  do  so. 

Brooks  (1993)  stated  that,  with  all  the  additional  capabilities  of  the  growing  number  of  multimedia 
applications,  the  design  of  these  applications  has  become  a nightmare.  He  also  pointed  out  the  preponderance  of 
ugly  interfaces  containing  screens  full  of  multiple  fonts,  insignificant  boxes,  irrelevant  noises,  and  confusing 
webs  of  possible  interactivity  among  the  features  of  poorly  designed  multimedia  packages.  There  are  many 
requirements  that  must  be  checked  while  designing  an  interface  such  as  screen  design,  learner  control  and 
navigation,  use  of  feedback,  student  interactivity,  and  video  and  audio  elements  (Stemler,  1997).  So,  the  design 
of  the  interface,  which  considers  interactivity,  is  clearly  important  (Frye  et  al.,  1988).  Hence  a properly 
designed  interface  should  make  the  cognitive  process  transparent  and  externalized  so  as  to  support  evaluation, 
reflection,  and  dis  cussion  and  direct  accessibility. 

The  following  principles  should  be  considered  during  the  design  of  ILEs  (Akpinar  & Hartley,  1996). 

• The  ILE  should  provide  interactive  objects  and  operators,  which  are  visual  and  can  be  directly 
manipulated  by  pupils. 

• The  ILE  system  should  provide  mechanisms  for  pupils  to  check  the  validity  of  their  methods, 
and  thus  receive  some  feedback  on  the  appropriateness  of  their  actions  in  relation  to  task. 

• As  the  instruction  aims  to  support  links  between  the  concrete  and  symbolic  representation  of 
word  problems,  the  ILE  should  be  able  to  display  these  forms  so  that  the  equivalence  between  is 
apparent.  The  system  should  also  be  able  to  move  its  presentation  modes  to  the  symbolic  as 
students  gain  in  competence. 

• The  ILE  should  allow  experimentation  of  concepts  and  procedures  in  ways  that  relate  to  the 
children’s  experiences.  In  brief  the  ILE  should  be  able  to  support  guided  discovery  as  well  as 
directed  methods  of  instruction. 

• The  ILE  should  allow  the  learning  to  be  conceptualized  and  procedural  in  its  approach,  and  be 
capable  of  adjusting  to  the  task  needs  of  teachers. 

Interactive  Learning  Environment:  LaborScale 

The  overall  aim  of  this  research  was  to  investigate  the  design  of  the  LaborScale  ILE  that  can  assist 
problem  solving  performance  and  understanding  specifically  mathematical  work  and  pool  problems.  Problem 
solving  requires  the  integration  and  utilization  of  multiple  knowledge  representations  e.g.  graphical,  symbolic, 
and  audio.  Depending  upon  these  factors,  the  design  of  the  computer  based  learning  environment  must  take  into 
consideration  students’  knowledge  so  that  it  can  accommodate  different  levels  of  competence  and  be  useful  for 
varying  modes  of  instruction  in  the  classroom(Mayer  1985).  The  proposed  design  to  realize  these  aims  is 
LaborScale  that  provides  a constructive  environment  based  on  direct  manipulation,  user-system  interactions  and 
available  interface  design. 

The  ILE  should  have  all  the  features  and  components  to  develop  children’s  word  problem  solving 
skills.  Hence,  it  should  be  designed  with  an  object-oriented  and  direct-manipulation  approach.  In  order  to  reach 
this  objective,  the  instructional  software  (LaborScale)  was  developed  and  implemented  by  the  researchers  using 
an  authoring  tool,  Asymetrix  Toolbook  II  5.0  and  other  related  multimedia  programs  (Macromedia  Flash  5.0, 

3D  Studio  Max  and  Photoshop  5.0)  to  support  Toolbook  Application  with  videos,  animations,  audios  and 
pictures.  The  user-friendliness  of  Toolbook  interface  and  its  accompanied  object-oriented  scripting  language, 
Openscript,  used  to  specify  the  functionality  were  well  suited  to  the  development  and  implementation  of  the 
prototype.  Toolbook  software  is  a development  environment  that  provides  tools  to  draw  objects  that  can  be 
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made  interactive  using  the  Openscript  programming  language.  Also  the  development  can  be  carried  out 
incrementally.  Further,  Toolbook  is  event-driven  i.e.  an  application  can  respond  to  events  such  as  mouse 
clicking  whenever  they  occur.  This  is  suitable  for  the  ILE  interface  that  is  based  on  a direct- manipulation 
approach  in  addition  to  its  ability  to  produce  quality  visual  animation. 

The  user-interface  of  LaborScale  has  two -page  design  consisting  of  multiple  viewers  in  which  each 
viewer  has  a background  and  a foreground  containing  objects  such  as  fields,  buttons,  graphics  and  text.  The  user 
interface  has  two  units:  a curriculum- manager  unit  and  a student-working  unit. 

Curriculum  Manager  Unit 

Curriculum  Manager  Unit  (CMU)  is  one  of  the  main  windows  of  the  LaborScale  (Figure  1).  This  is  the 
place  where  teachers  set  problems  and  customize  environments  for  students.  Each  problem  specification  will 
need  to  provide  context  information,  and  the  concepts  based  on  the  activity  sets.  Hence,  the  purpose  of  the 
problem  specification  is  to  provide  contexts  familiar  to  the  students. 

The  CMU  has  been  designed  to  manage  the  specification  of  activity  sets  that  contains  the  problem 
content,  specification  of  problems,  and  types  of  representation.  In  order  to  form  a new  activity  set  for  any 
student,  teachers  can  use  two  methods.  One  has  two  steps,  which  are  pre  -storing  problems  and  their  answers, 
and  specification  of  the  problems  depending  on  the  level  of  the  students  . The  other  is  only  specification  of 
problems,  which  are  saved  to  the  system  previously.  In  brief,  teachers  can  manage  the  following  tasks  by  using 
CMU: 


• Forming  problem  sets  including  simple  and  advanced  level  problems  of  two  types,  work  and 
pool,  by  saving  problems  to  the  systems , 

• Setting  audio  environment  of  student-working  unit, 

• Preparing  an  activity  set  with  respect  to  the  students’  level, 

• Looking  at  the  performances  of  the  students  who  finish  their  activity  sets , 

• Viewing  the  activity  set  in  the  student -working  unit. 

Figure  1 . A screen  of  curriculum  manager  unit 
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Student  Working  Unit 

The  student  working  window  (Figure  2),  the  learner  mode  of  the  LaborScale,  is  the  second  main 
window  of  the  LaborScale.  The  Curriculum-Manager  Unit  passes  the  sequence  of  problems  to  the  ILE 
controller  that  is  to  manage  the  interactions  with  students  and  to  keep  records  of  their  progress.  Hence  a 
principal  consideration  in  the  design  of  this  unit  was  the  user-system  interface  in  which  these  interactions  take 
place.  LaborScale  is  based  on  a high  degree  of  graphical  and  symbolic  object  manipulation,  and  with  the 
interface  users  are  able  to  directly  manipulate  the  LaborScale  objects,  for  example  by  giving  the  values 
symbolically,  dragging  and  dropping  of  picture  of  these  values,  and  combining  the  representations  of  them  to 
reach  a solution.  To  outline,  students  can  manage  the  following  tasks  in  this  unit: 


• Displaying  ratios  they  entered  in  the  problems , 

• Dragging  and  dropping  the  displayed  objects  and  displaying  a vertical  scale  as  a result  of  this , 

• Reaching  right  answers  of  the  problems  by  analyzing  a horizontal  scale  depending  on  the 
vertical  scale , 

• Setting  audio  environment, 

• Transition  to  other  problems. 

Figure  2.  A screen  of  student  working  unit 


Evaluation  studies 

The  validation  of  LaborScale  was  carried  out  during  last  week  of  May  and  first  week  of  June  2001. 
The  method  of  the  research  was  pretest  and  posttest  group  design.  The  sample  of  the  study  was  selected  from 
seventh  grade  students  of  two  different  schools  (Public  school,  school  A and  private  school,  school  B)  that  have 
a computer  laboratory.  The  subjects  were  selected  by  using  clustering  sample  technique,  namely,  one  class  was 
selected  from  each  school.  The  validation  experiment  was  performed  as  a pretest  of  the  work  and  pool 
problems.  The  pre-test  was  administered  to  80  students  (59  from  public  school  and  21  from  private  school). 


5 


Their  average  age  was  14  and  40  of  them  were  girls  and  40  of  them  were  boys.  For  the  application  and  post-test, 
convenient  students  were  chosen  from  each  class  by  considering  their  teacher’s  opinions  and  number  of 
computers  in  the  computer  laboratories  of  both  schools . Students  were  chosen  according  to  their  achievements 
for  each  mode  of  the  pre-test,  namely,  numerical  solutions,  symbolic  and  graphic  representations  of  solutions. 
The  students’  actions  in  LaborScale  were  recorded  by  the  system  and  notes  were  taken  by  the  researcher  as  well 
as  the  pre  and  posttest  differences  in  performance. 

The  instructor  organized  software  for  27  students  to  run  in  their  laboratories.  The  reason  for  choosing 
only  27  students  was  that  there  were  28  computers  in  the  computer  laboratory  of  School  A,  but  17  of  them  were 
available 

to  run  the  software.  There  were  13  computers  in  the  computer  laboratory  of  School  B,  but  10  of  them  were 
available  to  run  the  software.  Five  problems  randomly  chosen  from  nine  problems  were  assigned  to  these 
students  for  both  work  and  pool  problems  respectively  by  regarding  the  pre-test  scores  of  the  students.  Pool 
problems  were  given  to  them  at  the  second  week  of  the  application  to  allow  students  to  manage  problems  at 
ease  and  meaningfully.  For  low-achievers,  three  of  the  problems  were  simple  and  two  of  them  were  advanced. 
For  intermediate  students  and  high  achievers,  two  of  the  problems  were  simple  and  three  of  them  were 
advanced. 

Before  the  application  of  the  software  in  both  schools,  the  researcher  provided  students  with  an 
orientation  session  at  which  the  students  received  explanations  and  were  experienced  on  how  they  will  use  the 
program  by  presenting  worked  examples  for  solving  different  type  of  problems.  This  session  lasted  about  20 
minutes.  Since  the  students  in  both  schools  had  a regular  computer  course,  they  had  no  difficulty  in  controlling 
and  manipulating  the  environment  during  the  instruction. 

After  an  orientation  session,  all  groups  received  computer-assisted  treatment  for  two  hours  without  any 
break  for  two  weeks  respectively.  During  the  instruction,  students  were  left  alone  and  they  only  interacted  with 
computers.  They  solved  their  own  problems  about  the  program  by  themselves  except  system  problems  by  using 
the  help  and  information  modes  of  the  software.  Therefore,  the  researcher  behaved  like  an  observer  in  the 
application.  The  time  that  the  students  completed  their  activity  sets  changed  between  30  and  65  minutes  for 
work  problems,  however,  the  instruction  on  pool  problems  lasted  between  20  and  45  minutes. 

At  the  end  of  the  instruction  the  performance  test  (PT)  was  conducted  to  all  subjects  as  a post-test. 
After  all,  the  questionnaire  about  the  software  evaluation  was  applied  to  the  teachers  involved  in  the  study  to 
obtain  their  criticisms. 

In  order  to  analyze  the  differences  between  the  pre  and  post  tests  mean  scores  of  the  whole  group  and 
School  A obtained  from  each  mode  of  PT,  paired  sampled  t-test  was  used,  and  Wilcoxon  test  was  used  for 
school  B since  the  numbers  of  subjects  in  the  groups  were  too  small. 

The  pre  and  posttests  results  showed  significant  improvements  in  students’  performances  for  each 
mode  that  pointed  to  the  benefits  of  LaborScale  ILE  (See  Table  1,  Table  2,  and  Table  3).  When  the  schools 
were  analyzed  separately,  there  was  a significant  increase  in  all  modes  of  the  tests  for  each  school.  Looking  at 
the  posttest  results  of  each  mode,  significant  improvements  were  also  observed  (for  more  details  see  Adiguzel, 
2001). 


Table  1 Differences  between  pre-post  test  scores  of  numerical  solutions  to  PT 


Groups 

N 

Pre-test 
Num.  Sol. 
Mean 

Pre-test 
Num.  Sol. 
Std.  Dev. 

Post-test 
Num.  Sol. 
Mean 

Post-test 
Num.  Sol. 
Std.  Dev. 

t 

Sig. 

(2 -tailed) 

Total 

27 

22.85 

27.28 

85.78 

21.70 

10.997* 

.000 

School  A 

17 

18.47 

25.36 

88.35 

24.43 

9.036* 

.000 

z 

School  B 

10 

30.30 

30.16 

81.40 

16.29 

2.805* 

.005 

*p  < .05. 

Conclusion 

Studies  formed  a base  for  using  computers  featuring  multiple  linked  representations  to  assist  students 
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with  the  transition  from  concrete  experiences  to  abstract  mathematical  ideas,  with  the  practice  of  skills,  and  with 
the  process  of  problem  solving,  like  in  the  LaborScale  ILE,  namely,  beginning  with  the  concrete  representations 
and  reaching  the  symbolic  representations  by  using  visual  components  supported  by  audio  developed  seventh 
grade  students’  performance  on  work  and  pool  problems. 
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Table  2 Differences  between  pre-post  test  scores  of  symbolic  mode  of  PT 


Groups 

N 

Pre-test 

Symbolic 

Mean 

Pre-test 
Symbolic 
Std.  Dev. 

Post-test 

Symbolic 

Mean 

Post-test 
Symbolic 
Std.  Dev. 

t 

Sig. 

(2 -tailed) 

Total 

27 

2.74 

8.86 

92.37 

11.27 

36.443* 

.000 

School  A 

17 

2.47 

9.68 

96.59 

5.49 

37.343* 

.000 

z 

School  B 

10 

3.20 

7.73 

85.20 

14.92 

2.807* 

.005 

*p  < .05. 

Table  3 Differences  between  pre-post  test  scores 

of  graphic  mode  of  PT 

Groups 

N 

Pre-test 

Graphic 

Mean 

Pre-test 
Graphic 
Std.  Dev. 

Post-test 

Graphic 

Mean 

Post-test 
Graphic 
Std.  Dev. 

t 

Sig. 

(2 -tailed) 

Total 

27 

18.67 

18.62 

82.26 

31.25 

10.380* 

.000 

School  A 

17 

21.65 

14.71 

95.00 

10.90 

16.858* 

.000 

z 

School  B 

10 

13.60 

23.92 

60.60 

42.21 

2.499* 

.012 

*p  < .05. 

In  the  post-test,  students  had  different  graphical  representations  of  solutions.  The  results  showed  that 
they  were  affected  from  the  visual  components  of  the  LaborScale  ILE,  namely,  some  of  them  drew  a box,  meant 
whole  work,  for  each  worker  and  they  indicated  the  results  by  the  graph  corresponding  these  related  graphs. 
Some  of  them  drew  a vertical  scale  whose  pointer’s  location  showed  the  values  of  work  done  in  a day  for  each 
worker  and  the  result.  The  rest  of  the  students  drew  a horizontal  scale,  similar  to  the  number  line,  whose 
pointer’s  location  also  represented  the  values  of  work  done  in  a day  for  each  worker  and  the  result.  This  proves 
that  if  the  students  are  given  more  visual  representations,  they  will  use  and  connect  them  to  the  symbolic 
representation  of  these  and  they  will  grasp  the  meaning  of  the  word  problem  solving  by  concretizing  them. 

The  increase  in  the  symbolic  mode  was  more  than  the  other  modes.  This  proves  that  multiple  linked 
representations  relating  the  symbolic  representation  to  graphic  representation  allow  learners  to  perceive 
complex  ideas  in  a new  way  and  to  apply  them  more  effectively.  However,  the  increase  in  the  graphical  mode 
was  significant  but  less  than  the  other  modes.  The  reason  for  this  may  be  that  since  text -based  books  were 
dominant  in  the  curriculum  children  have  not  developed  to  present  problems  graphically.  However,  the 
graphical  representations  need  to  be  well  constructed  and  be  capable  of  representing  the  information  in  a 
problem  to  enable  the  processing  capabilities  of  the  human  visual  system  to  be  exploited,  so  that  perceptual 
features  and  judgments  can  be  developed  and  related  to  a more  abstract  symbolic  understanding  (Cox  & Bma, 
1995).  Also,  graphical  representations  are  effective  problem  solving  and  learning  tools  because  they  reduce  the 
space  of  applicable  operations  and  they  are  more  specific  than  the  other  representations.  Relating  with  the 
theory,  good  performance  on  finding  and  presenting  a graphic  representation  of  a solution  raised  the 
performance  on  grasping  symbolic  representation  of  the  solution  in  the  study. 

A similar  result  to  the  findings  of  this  research  was  the  outcome  of  ANIMATE  software  (Nathan, 

1991).  However,  the  potentials  and  facilities  of  ANIMATE  d iffer  from  the  LaborScale.  Though  Nathan’s 
ANIMATE  can  not  generalize  the  solution  method  into  an  algebraic  formula,  LaborScale  can  help  students  to 
build  up  algebraic  formula  that  may  be  generalized  and  employed  in  a wide  variety  of  problems 
During  the  application,  the  time  at  which  students  finished  each  activity  set  was  recorded  by  the  researcher. 
According  to  the  results,  students  spent  longer  time  period  on  the  first  question  of  the  activity  set  of  the  work 
problems  than  the  other  questions  of  the  activity  set  since  they  were  adapted  to  the  system  in  the  first  question. 
After  they  gained  an  experience  on  the  work  problems,  they  acted  carefully  and  swiftly  on  the  pool  problems 


since  the  application  of  the  pool  problems  was  applied  to  them  one  week  later. 

Performance  recording  unit  of  the  program  stored  two  main  functions  of  the  students:  three  trials  on 
the  values  that  are  work  done  in  a day  for  each  worker  and  the  answer  of  the  problem  for  each  trial.  According 
to  the  records  obtained  from  this  unit,  all  students  had  tried  at  least  three  times  on  the  advanced  level  problems 
and  problems  requiring  complex  calculation  to  reach  a right  answer.  The  reason  for  this  was  the  greatness  of  the 
least  common  multiple  related  with  the  value  of  denominator  of  the  work  done  in  one  day  and  that  because  of 
the  big  unit  differences  in  the  scale  students  could  not  grasp  net  measurement.  However,  they  usually  solved 
these  problems  after  they  tried  at  least  three  times.  The  main  thing  in  this  part  is  requiring  the  students  to  grasp 
the  values  from  the  problem.  As  a result,  students’  problem  solving  skills  significantly  improved  by  the  help  of 
instructional  software,  LaborScale . 

The  rationale  of  the  development  of  the  LaborScale  was  to  base  students’  problem  solving  on  multiple 
representations.  This  should  aid  understanding,  conform  to  problem  solving  as  an  investigatory  and  creative 
activity.  While  LaborScale  was  successful  in  fulfilling  many  of  these  claims  in  its  design  and  conception,  there 
are  further  requirements  to  complement  or  supplement  its  current  facilities  and  strengths. 

In  conclusion,  the  LaborScale  software  and  validation  studies  have  given  some  support  to  the  design 
principles  of  ILEs  in  which  the  aim  is  to  produce  user-system  interfaces  that  release  students’  knowledge  and 
stimulate  active  and  investigatory  methods  of  learning.  The  suggested  further  work  could  re  -illuminate  design 
principles  for  multi-representational  interfaces.  And,  since  the  research  was  concluded  successfully  and 
significant  results  were  obtained,  hopefully  it  should  be  adapted  for  other  domains  of  mathematics. 
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Investigating  the  Relationships  Among  Instructional  Strategies  and 
Learning  Styles  in  Online  Environments 

Omur  Akdemir 
Tiffany  A.  Koszalka 
Syracuse  University 

Abstract 

Researchers  investigated  differences  in  learner  preferences  for  different  types  of  instructional 
strategies  and  learning  styles  in  online  environments.  Results  suggested  that  matches  between  students’ 
learning  styles  and  instructional  strategies  did  not  affect  their  perception  of  their  own  learning  outcomes,  level 
of  effort  and  involvement,  and  level  of  interactions  in  the  course.  Data  also  indicated  that  no  single 
instructional  strategy,  among  three  instructional  strategies  tested,  emerged  as  superior  for  high  and  low  field 
dependent  online  students. 


Introduction 

The  Internet  has  taken  center  stage  today  as  a preferred  medium  for  the  delivery  of  distance  education. 
Many  universities  offer  online  courses  that  respond  to  the  diverse  distance  and  time  needs  of  today’s  learners. 
These  universities  provide  course  instructors  with  online  tools  to  manage  course  participation  and  facilitate 
learning.  Instructors  can  continuously  monitor  student  progress,  provide  learners  with  time  to  reflect  on  content 
and  feedback  before  participating,  prompt  active  participation  with  content  and  peers,  and  offers  instructional 
modules  that  are  designed  to  appeal  to  a variety  of  learning  styles  and  preferences  (Hamilton-Pennell,  2002). 

Learning  style  can  be  thought  of  as  the  combination  of  the  learners’  motivation,  task  engagement,  and 
information-processing  habits  (Aragon,  Johnson,  & Shaik,  2002).  Each  learner  can  have  different  preferences  as 
to  how  s/he  receives,  processes,  and  recalls  information  during  instruction.  Many  researchers  however,  have  not 
controlled  for  students’  characteristics  in  their  analyses  of  students’  satisfaction  of  online  instruction 
(Thurmond,  Wambach,  & Connors,  2002).  Understanding  the  relationships  among  learning  styles  and 
instructional  preferences  holds  great  promise  for  enhancing  educational  practice  (Claxton  & Murrell,  1987). 

The  primary  purpose  of  this  exploratory  pilot  study  was  to  investigate  the  relationships  among  learning 
styles,  defined  as  high  and  low  field  dependence,  and  preferences  for,  and  evaluation  of,  instructional  strategies 
used  in  an  online  course.  Field  dependence  describes  the  degree  to  which  a learner’s  perception  or 
comprehension  of  information  is  affected  by  the  surrounding  contextual  field  (Jonassen  & Grabowski,  1993). 
Learning  styles  are  useful  because  they  provide  information  about  individual  differences  from  a cognitive  and 
information-processing  standpoint  (Smith  & Ragan,  1999).  Field  dependent  individuals  are  more  likely  to 
succeed  at  learning  tasks  that  engage  them  in: 

Group  oriented  and  collaborative  work  situations 

Situations  where  individuals  have  to  follow  standardized  pattern  of  performance 

Tests  requiring  individuals  to  recall  information  in  the  form  or  structure  that  it  was  presented 

(Jonassen  & Grabowski,  1993). 

High-field  dependent  individuals  have  more  difficulty  locating  the  information  they  are  looking  for 
than  low  field  dependent  individuals.  Low  field  dependent  individuals  are  more  likely  to  excel  at  learning  tasks 
involving  identification  of  important  aspects  of  information  from  a poorly  organized  body  of  information.  High 
field  dependent  individuals  tend  to  accept  the  information  without  reorganizing  it  from  the  way  it  was  presented 
to  them  so  low  field  dependent  individuals  are  likely  to  reorganize  information  to  fit  their  own  perceptions. 

Muir  (2001)  recommends  teaching  methods  that  match  instructional  strategies  to  field  dependence- 
independence  style. 

Instructional  strategies  represent  a set  of  decision  that  result  in  plan,  method,  or  series  of  activities 
aimed  at  obtaining  a specific  goal  (Jonassen,  Grabinger,  & Harris,  1990).  Instructional  strategies  are  the 
activities  used  to  engage  learners  in  the  learning  process.  Many  types  of  instructional  strategies  are  used  to 
engage  learner  in  different  ways  such  as  reading,  collecting,  thinking,  etc.  Expository  strategies  may  include 
providing  learners  with  lecture  notes.  Explanations  are  often  kept  simple  and  direct.  Students  usually  use  lecture 
notes  to  complete  learning  activities  or  respond  to  posed  questions.  Collaborative  and  group  work  instructional 
strategies  require  individuals,  often  at  various  levels,  to  work  together  to  achieve  a common  goal.  Individuals 
are  prompted  to  analyze,  synthesize,  and  evaluate  their  ideas  collaboratively.  Inquisitive  (discovery  learning) 
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instructional  strategies  require  individuals  to  formulate  investigative  questions,  obtain  factual  information,  and 
build  knowledge,  which  reflects  their  answer  to  the  original  question.  Students  develop  several  questions,  which 
eventually  lead  them  to  answer  the  original  question,  use  extensive  resources  to  gather  data,  and  answer  the 
original  question. 

The  characteristics  of  high  field  dependent  individuals  appear  to  match  with  expository  (presentation), 
and  collaborative  (group  work)  types  of  strategies  because  these  types  of  instructional  strategies  require  learners 
to  complete  learning  activities  that  are  usually  kept  simple,  and  sometimes  require  learners  to  work  together. 

The  characteristics  of  low  field  dependent  individuals  suggest  a match  with  inquisitive  type  of  strategies 
because  low  field  dependent  individuals  prefer  generating  their  own  hypothesis  and  testing  their  hypothesis. 
Table  1 illustrates  the  suggested  match  and  mismatch  of  learning  style  and  instructional  strategy  for  this  study. 


Table  1 .Match  and  Mismatch  of  Learning  Style  and  Instructional  Strategy 


Expository 

Collaborative 

Discovery 

High  Field  Dependent 

Match 

Match 

Mismatch 

Low  Field  Dependent 

Mismatch 

Mismatch 

Match 

Abraham  (1985)  found  that  matching  instructional  styles  to  students’  field-dependent  or  independent 
style  improved  students’  performance  in  the  course.  In  the  study,  researchers  used  two  computer-assisted 
instruction  lessons,  one  rule  oriented,  and  the  other  deemphasizing  rules,  to  test  whether  a teaching  approach 
that  did  not  emphasize  rules  would  be  of  greater  benefit  to  field-dependent  students  in  an  English  as  a second 
language  class.  The  results  of  the  study  showed  that  field-independent  students  performed  better  with  rule 
oriented  approach  whereas  field-dependent  students  performed  better  with  the  approach  deemphasizing  rules. 
There  has  also  been  research  that  was  contradictory  to  these  results.  Macneil  (1980)  found  that  learning  did  not 
increase  when  students  categorized  as  field  dependent  and  field  independent  receive  instruction  oriented  to  their 
style.  In  the  study,  researchers  used  discovery  and  expository  approaches  to  test  whether  randomly  assigned 
field  dependent  students  learn  more  from  the  discovery  approach  and  field  independent  students  learn  more 
from  expository  approach.  Results  of  the  study  revealed  that  achievement  of  field  dependent  and  field 
independent  students  did  not  vary  as  a function  of  style.  The  question  remains  can  matching  learning  styles  and 
instructional  strategies  in  distance  education  better  support  student  learning.  This  study  was  designed  to  address 
the  following  research  questions: 

1.  Is  there  a difference  in  perceived  learning  outcomes  for  students  whose  learning  style  matches  with  the 
instructional  strategy? 

2.  Is  there  a difference  in  students’  effort  and  involvement  for  students  whose  learning  style  matches  with 
the  instructional  strategy? 

3.  Is  there  a difference  in  students’  perceived  level  of  interaction  for  students  whose  learning  style 
matches  with  the  instructional  strategy? 

4.  Is  there  a difference  in  perceived  learning  outcomes  for  low  field  dependent  learners  in  match  and 
mismatch  instructional  strategy  situations? 

5.  Is  there  a difference  in  perceived  learning  outcomes  for  high  field  dependent  learners  in  match  and 
mismatch  instructional  strategy  situations? 

Method 

Instructional  Context 

The  pilot  study  was  conducted  at  a private  university  located  in  the  northeastern  United  States  with 
graduate  students  enrolled  in  an  online  graduate  course  entitled  Design  and  Management  of  Distance 
Education.  This  investigation  focused  on  determining  if  students  who  were  classified  as  low  or  high  field 
dependent  perceived  different  types  of  instructional  strategies  differently  in  an  online  instructional  environment. 
Specifically,  students  would  be  queried  about  their  perceptions  of  learning  outcomes,  their  effort  and 
involvement  in  the  activities,  and  their  level  of  interaction  during  the  course. 

The  Design  and  Management  of  Distance  Education  course  consisted  of  three  modules.  Each  module 
was  delivered  online  using  a different  instructional  strategy  including,  expository  (presentation),  collaborative 
(group  work),  and  inquisitive  (discovery  learning).  All  three  units  were  experiential  and  generative  in  nature, 
requiring  learners  to  interact  in  different  ways  with  the  content  to  facilitate  learning.  On  average,  each  unit  was 
completed  over  a four-week  period. 

Expository  type  of  instructional  strategy  was  utilized  primarily  to  present  module  one  content.  Each 
student  read  the  assigned  chapters  in  the  course  text,  specified  web  pages,  and  powerpoint  slides  regarding  the 
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growth  and  development  of  the  field  of  distance  education.  Students  were  then  required  to  participate 
asynchronous  discussions  responding  to  initial  question  posted  by  course  professor  and  at  least  two  other 
postings  from  their  peers  supporting  their  responses  with  references  from  readings.  Finally  students  were 
required  to  write  a reflection  journal  and  complete  content  quizzes. 

Module  two  was  presented  using  collaborative  group  work.  Four  teams  of  3 to  4 students  were 
established.  A case  scenario  was  presented  and  each  team  was  asked  to  design  a prototype  distance  education 
course  based  on  specified  criteria.  A private  discussion  forum  and  workspace  was  made  available  to  each  team 
to  support  their  collaboration  while  completing  the  module.  Throughout  the  module,  each  team  was  expected  to 
submit  status  reports,  and  a final  instructional  design  report.  Quality  of  the  deliverables  and  level  of 
participation  were  used  as  evaluation  criteria. 

Inquisitive  (discovery  learning)  types  of  strategies  were  used  to  present  module  three.  Students  were 
prompted  to  explore  methods,  media,  and  materials  in  distance  education,  to  identify  most  important  points  of 
their  implementation,  and  to  prepare  a mini  presentation  describing  benefits  and  challenges  of  each.  In  addition 
to  the  course  text,  and  additional  web  links,  students  were  expected  to  utilize  other  resources  to  prepare  the  mini 
presentation.  Then,  students  were  expected  to  participate  in  a bulletin  board  discussion,  write  a reflection 
journal  describing  the  at  least  five  web  sources  helping  them  to  better  understand  on  hot  topic  in  distance 
education  related  to  methods,  media,  or  materials.  For  example,  if  a student  was  curious  about  copyright  s/he 
would  explore  the  topic  and  report  findings  back  to  class.  Ultimately,  students  were  prompted  to  respond  to 
inquiries  into,  and  learn  about  distance  education  by  investigating  a variety  of  distance  education  areas  of  their 
own  choice,  and  share  their  findings  with  the  class. 

Subjects 

The  subjects  included  twelve  graduate  students  registered  for  this  course.  Sixty-six  percent  of  the 
students  were  doctoral  students  and  others  were  master  degree  students.  Four  students  reported  their  technical 
skill  as  advanced.  The  other  eight  studied  described  their  technical  skills  as  intermediate.  Sixty-six  percent  of 
the  students  had  taken  at  least  one  online  course  before  enrolling  in  this  course.  The  results  of  the  Psychological 
Differentiation  Inventory  showed  that  25%  of  the  students  were  high  field  dependent  and  others  were  low  field 
dependent  students. 

Instruments 

In  order  to  conduct  this  research  a valid  and  reliable  measure  of  learning  style  had  to  be  secured  that 
could  be  implemented  online.  One  such  measure  used  for  decades  to  study  learning  styles  is  the  Group 
Embedded  Figures  Test  (GEFT)  (Witkin  et  al.,  1971).  The  GEFT  is  used  for  measuring  field  dependence  and 
independence.  However,  the  use  of  this  instrument  is  problematic  for  online  environments  because  of  the 
requirement  to  time  participant  responses  and  because  participants  have  to  draw  responses  in  a given  booklet. 
Given  that  distributed  nature  of  students,  the  reliability  of  each  participant  completing  the  instrument  per 
instruction  is  questionable.  Therefore,  the  investigator  searched  for  a version  of  the  instrument  that  could  be 
implemented  online.  The  Psychological  Differentiation  Inventory  (PDI),  a questionnaire  measure  of  field 
dependence  was  reconstructed  as  an  online  questionnaire  for  this  study  and  used  to  measure  high  field 
dependence  and  low  field  dependence  of  participating  learners.  The  PDI  has  good  test-retest  reliability  (.69) 
and  correlates  (r  = 0.  46  - 0.76)  with  Embedded  Figure  Test  which  is  frequently  used  as  a single  measure  of 
field  dependence  (Evans,  1969). 

In  this  research  the  evaluation  system  used  to  assess  students’  achievements  in  each  module  included 
three  components.  These  components  were  (1)  self-assessment  of  outcome,  (2)  individual  effort  and 
involvement,  and  (3)  interaction  and  feedback  between  and  among  the  instructor  and  students  (Robles  & 
Braathen,  2002).  The  modified  version  of  Student  Instructional  Report  II  developed  by  John  A.  Centra  in  1998 
was  used  with  permission  to  assess  components  1 and  2.  This  instrument  contains  five  items  for  assessing 
perceived  unit  outcome  of  students,  and  three  items  for  assessing  student  effort  and  involvement.  Returns 
indicated  the  student’s  perception  of  the  effectiveness  of  each  aspect  of  a unit  to  the  same  aspects  in  other  units 
using  a five-point  scale.  A rubric  developed  by  Roblyer  & Wiencke  in  2003  was  used  to  assess  the  level  of 
interactivity  in  each  module  by  having  students  evaluate  elements  of  interactions  including  social  goals, 
instructional  goals,  types  and  uses  of  technology,  and  impact  of  interactivity-changes  in  learner  behaviors. 

Procedure 

The  Design  and  Management  of  Distance  Education  course  consisted  of  three  modules.  Each  module 
had  to  be  completed  in  order,  and  in  a given  time  frame  by  all  students.  Data  were  collected  after  each  unit  was 
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completed.  The  online  unit  evaluation  form  at  the  end  of  each  unit  measured  learner  satisfaction  and 
involvement  with  the  instruction  specifically  through  (1)  perceived  unit  outcomes,  (2)  student  perception  of 
effort  and  involvement  in  the  unit,  and  (3)  student  perception  of  interaction  and  feedback  levels  between  and 
among  the  instructor  and  students  during  the  unit  (Roblyer  & Wiencke,  2003;  Centra,  1998).  A java  script  was 
written  for  the  online  unit  evaluation  form  to  ensure  that  students  answered  all  questions  before  submitting  it. 
Using  java  script  eliminated  the  risk  of  missing  question  response.  Upon  completing  the  online  unit  evaluation 
form,  the  data  were  automatically  emailed  to  the  researchers. 

Students  also  had  to  complete  the  online  questionnaire  version  of  the  Psychological  Differentiation 
Inventory  to  measure  their  level  of  field  dependence.  A java  script  was  also  written  for  the  online  questionnaire 
version  of  the  Psychological  Differentiation  Inventory  to  ensure  that  students  answered  all  questions  on  the 
inventory.  Researchers  also  received  the  results  of  the  Psychological  Differentiation  Inventory  through  email. 

Analysis 

All  data  were  ported  into  a statistical  analysis  package  (Stata  version  8.0)  for  later  analysis.  One  way 
analysis  of  variance  was  used  to  test  the  hypotheses  that  there  were  differences  in  students  perceived  learning 
outcomes,  students  effort  and  involvement,  and  students’  perceived  level  of  interaction  when  students  learning 
style  matches  with  the  instructional  strategy,  and  to  test  whether  one  instructional  strategy  emerges  with  higher 
perceived  learning  outcomes  for  online  students  who  are  categorized  as  high  field  dependent  and  low  field 
dependent.  All  statistical  analysis  reported  in  this  research  were  conducted  with  a significant  level  of  .05. 

Results 

Learning  style 

The  results  of  the  online  questionnaire  version  the  Psychological  Differentiation  Inventory  revealed 
that  nine  students  were  low  field  dependent  and  three  students  were  high  field  dependents.  The  mean  score  for 
students  categorized  as  low  field  dependent  was  19.55  (S.D.  = 3.53)  while  the  mean  score  for  students 
categorized  as  high  field  dependent  was  26.33  (S.D.  = 0.57)  (see  Table  2). 


Table  2.  Means  and  Standard  Deviations  for  Students  Categorized  as  Low  Field  Dependent  and  High  Field 
Dependent 


Categories 

N 

Mean 

Standard 

Deviation 

Min 

Max 

Low  Field 
Dependent 

9 

19.55 

3.53 

14 

23 

High  Field 
Dependent 

3 

26.33 

0.57 

26 

27 

Matching  Learning  Style  with  Instructional  Strategy 

The  first  hypothesis  stated  that  there  would  be  no  significant  difference  in  the  perceived  learning 
outcomes  of  students  whose  learning  style  matched  the  instructional  strategy.  The  results  of  the  one-way 
analysis  of  variance  supported  this  null  hypothesis,  F (2,18)  = 0.11,  p = 0.89  (see  Table  3).  No  significant 
difference  was  found  in  the  perceived  learning  outcomes  of  students  whose  learning  style  matched  the 
instructional  strategy.  Both  low  and  high  field  dependent  students  perceived  learning  outcomes  in  the  three 
instructional  strategies  the  same.  Table  4 shows  the  descriptive  statistics  for  perceived  learning  outcomes  of 
students  whose  learning  style  matched  the  instructional  strategy. 


Table  3.  Results  of  One-way  Analysis  of  Variance  for  Perceived  Learning  Outcomes  of  Students  whose 
Learning  Style  Matched  the  Instructional  Strategy  Used  to  Present  the  Online  Course  Module 


Source 

Sum  of  Squares  D.F. 

Mean  Squares 

F ratio 

F Prob. 

Between  groups 
Within  groups 
Total 

.18031733  2 

14.4977771  18 

14.6780945  20 

.090158665 

.805432064 

.733904724 

0.11 

0.8947 

Table  4.  The  Descriptive  Statistics  for  Perceived  Learning  Outcomes  of  Students  whose  Learning  Style  Matched 
the  Instructional  Strategy 

Matched  Group 

Instructional  Strategy  Mean  N 

S.D. 

Min  Max 

14 


Low  Field  Dependent 

Expository 

3.71 

9 

0.85 

2 

5 

Low  Field  Dependent 

Collaborative 

3.55 

9 

0.88 

2 

5 

High  Field  Dependent 

Discovery 

3.46 

3 

1.1 

2.4 

4.6 

The  second  hypothesis  stated  that  there  would  be  no  significant  difference  in  the  effort  and 
involvement  of  students  whose  learning  style  matched  the  instructional  strategy  used  to  present  the  online 
course  module.  The  results  of  the  one  way  analysis  of  variance  supported  this  null  hypothesis,  F(2,18)  = 1.02 ,p 
= 0.37  (see  Table  5).  No  significant  difference  was  found  in  the  effort  and  involvement  of  students  whose 
learning  style  matched  the  instructional  strategy  used  to  present  the  online  course  module.  When  low  and  high 
field  dependent  students’  learning  styles  matched  three  types  of  instructional  strategies  used  in  the  study,  low 
and  high  field  dependent  students  reported  they  put  equal  effort  and  involvement  to  instructional  activities. 
Table  6 shows  the  descriptive  statistics  for  the  effort  and  involvement  of  students  whose  learning  style  matched 
the  instructional  strategy. 

Table  5.  Results  of  One  way  Analysis  of  Variance  for  Effort  and  Involvement  of  Students  whose  Learning  Style 
Matched  the  Instructional  Strategy  Used  to  Present  the  Online  Course  Module 


Source 

Sum  of  Squares 

D.F. 

Mean  Squares 

F ratio 

F Prob. 

Between  groups 

1.06779522 

2 

.533897609 

1.02 

0.3795 

Within  groups 

9.39358058 

18 

.521865588 

Total 

10.4613758 

20 

.52306879 

Table  6.  The  Descriptive  Statistics  for  the  Effort  and  Involvement  of  Students  whose  Learning  Style  Matched  the 
Instructional  Strategy 


Matched  Group 

Instructional  Strategy 

Mean 

N 

S.D.  Min 

Max 

Low  Field  Dependent 

Expository 

3.71 

9 

0.  5 3 

4.4 

Low  Field  Dependent 

Collaborative 

3.81 

9 

0.64  3 

5 

High  Field  Dependent 

Discovery 

3.13 

3 

1.41  1.6 

4.4 

The  third  null  hypothesis  stated  that  there  would  be  no  significant  difference  in  the  perceived  level  of 
interaction  of  students  whose  learning  style  matched  the  instructional  strategy.  The  results  of  the  one  way 
analysis  of  variance  supported  this  hypothesis,  F(2,18)  = 0.03,  p = 0.97  (see  Table  7).  No  significant  difference 
was  found  in  the  perceived  level  of  interaction  of  students  whose  learning  style  matched  the  instructional 
strategy.  Low  and  high  fie  Id  dependent  students  perceived  their  level  of  interactivity  same  for  all  three  types  of 
instructional  strategies  used  in  these  modules.  Table  8 shows  the  descriptive  statistics  for  the  level  of  interaction 
perceived  by  students  whose  learning  style  matched  the  instructional  strategies. 

Table  7.  Results  of  One-way  Analysis  of  Variance  for  Perceived  Level  of  Interaction  of  Students  whose 
Learning  Style  Matched  the  Instructional  Strategy  Used  to  Present  the  Online  Course  Module 


Source 

Sum  of  Squares 

D.F. 

Mean  Squares 

F ratio 

F Prob. 

Between  groups 

.054603198 

2 

.027301599 

0.03 

0.9703 

Within  groups 

16.2755553 

18 

.904197518 

Total 

16.3301585 

20 

.816507926 

Table  8.  The  Descriptive  Statistics  for  the  Level  of  Interaction  Perceived  by  Students  whose  Learning  Style 
Matched  the  Instructional  Strategies 


Matched  Group 

Instructional  Strategy 

Mean 

N 

S.D. 

Min 

Max 

Low  Field  Dependent 

Expository 

3.82 

9 

0.92 

2.6 

5 

Low  Field  Dependent 

Collaborative 

3.77 

9 

0.95 

2.66 

5 

High  Field  Dependent 

Discovery 

3.66 

3 

1.0 

2.6 

4.6 

One  Superior  Instructional  Strategy 

The  fourth  null  hypothesis  stated  that  there  would  be  no  significant  difference  in  the  perceived  learning 
outcomes  for  low-field -dependent  learners  in  match  and  mismatch  instructional  strategy  situations.  The  results 
of  the  one  way  analysis  of  variance  supported  this  null  hypothesis,  F( 2,24)  = 0.19,  p = 0.82  (see  Table  9).  No 
significant  difference  was  found  in  the  perceived  learning  outcomes  of  low- field-dependent  students  who 
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completed  three  online  course  modules.  The  characteristics  of  low  field  dependent  students  showed  match  with 
expository  and  collaborative  type  of  instructional  strategies,  and  mismatch  with  discovery  type  of  instructional 
strategies.  Statistical  analysis  showed  no  s ignificant  difference  in  the  perceived  learning  outcomes  of  low  field 
dependent  students  in  match  and  mismatch  instructional  strategy  situations.  Table  10  shows  the  descriptive 
statistics  for  the  perceived  learning  outcomes  for  low  field  dependent  learners  in  match  and  mismatch 
instructional  strategy  situations. 

Table  9.  Results  of  One-way  Analysis  of  Variance  for  Perceived  Learning  Outcomes  of  Low-Field-Dependent 
Students  in  Match  and  Mismatch  Instructional  Strategy  Situations 


Source 

Sum  of  Squares 

D.F. 

Mean  Squares 

F ratio 

F Prob. 

Between  groups 

.234073991 

2 

.117036996 

0.19 

0.8286 

Within  groups 

14.8266668 

24 

.617777781 

Total 

15.0607407 

26 

.579259259 

Table  10.  The  Descriptive  Statistics  for  the  Perceived  Learning  Outcomes  for  Low  Field  Dependent  Learners  in 
Match  and  Mismatch  Instructional  Strategy  Situations 


Low  Field  Dependent 

Learners  Match  and  Mismatch  Situations 

Instructional  Strategy 

Mean 

N 

S.D. 

Min 

Max 

Match 

Expository 

3.71 

9 

0.85 

2 

5 

Match 

Collaborative 

3.55 

9 

0.88 

2 

5 

Mismatch 

Discovery 

3.77 

9 

0.58 

3 

4.8 

The  last  null  hypothesis  stated  that  there  would  be  no  significant  difference  in  the  perceived  learning 
outcomes  for  high-field-dependent  learners  in  match  and  mismatch  instructional  strategy  situations.  The  results 
of  the  one  way  analysis  of  variance  supported  this  null  hypothesis,  F{ 2,6)  = 0.13,/?  = 0.88  (see  Table  11).  No 
significant  difference  was  found  in  the  perceived  learning  outcomes  of  high- field-dependent  students  who 
completed  three  online  course  modules  each  of  which  used  different  instructional  strategy.  Perceived  learning 
outcomes  of  high  field  dependent  students  did  not  change  when  they  were  taught  with  different  instructional 
strategies  matching  and  mismatching  their  characteristics.  Table  12  shows  the  descriptive  statistics  for  the 
perceived  learning  outcomes  for  high  field  dependent  learners  in  match  and  mismatch  instructional  strategy 
situations. 

Table  11.  Results  of  One-way  Analysis  of  Variance  for  Perceived  Learning  Outcomes  of  High-Field-Dependent 
Students  in  Match  and  Mismatch  Instructional  Strategy  Situations 


Source 

Sum  of  Squares 

D.F. 

Mean  Squares 

F ratio 

F Prob. 

Between  groups 

.267654316 

2 

.133827158 

0.13 

0.8824 

Within  groups 

6.28740728 

6 

1.04790121 

Total 

6.5550616 

8 

.8193827 

Table  12.  The  Descriptive  Statistics  for  the  Perceived  Learning  Outcomes  for  High  Field  Dependent  Learners 
in  Match  and  Mismatch  Instructional  Strategy  Situations 


High  Field  Dependent 

Learners  Match  and  Mismatch  Situations 

Instructional  Strategy 

Mean 

N 

S.D. 

Min 

Max 

Mismatch 

Expository 

3.66 

3 

0.94 

2.6 

4.4 

Mismatch 

Collaborative 

3.88 

3 

1.01 

3 

5 

Match 

Discovery 

3.46 

3 

1.1 

2.4 

4.6 

Discussion  and  Conclusion 

Delivering  instruction  on  the  Internet  has  become  very  popular  in  recent  years.  Often  face-to-face 
courses  are  converted  to  online  course  activities  and  materials  with  little  thought  of  learners’  preferences  for 
instruction.  Understanding  the  effects  that  learning  styles  and  learners’  perceptions  of  engagement  in  online 
environments  have  potential  to  improve  the  planning,  producing,  and  implementing  of  online  educational 
experiences.  Thus,  learning  styles  can  be  utilized  to  enhance  students’  learning,  retention,  and  retrieval 
(Federico,  2000).  This  study  provides  insight  into  the  relationships  among  learning  style  and  instructional 
strategies  used  in  online  environments. 
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The  statistical  analysis  revealed  no  significant  differences  among  three  match  situations  for  low  and 
high  field  dependent  students.  When  the  characteristics  of  low  and  high  field  dependent  students  matched  with 
instructional  strategies,  match  groups  did  not  show  any  statistically  significant  difference  in  their  perceived 
learning  outcomes,  their  perceived  effort  and  involvement  in  units,  and  level  of  interactivity  that  they  perceived 
during  the  unit.  This  result  showed  that  when  low  and  high  field  dependent  students  receive  instruction  utilizing 
instructional  strategies  matching  their  characteristics,  they  gain  equal  learning  benefits  from  the  instruction. 
Using  expository  and  collaborative  type  of  instructional  strategies  for  high  field  dependent  students,  and  using 
discovery  type  of  instructional  strategies  for  low  field  dependent  students  in  online  courses  provided  equal 
benefits  for  students  in  terms  of  their  perceived  learning  outcomes,  their  perceived  effort  and  involvement,  and 
level  of  interactivity  that  they  perceived  in  the  class.  However,  considering  the  fact  that  mean  scores  of  students 
for  match  situations  were  more  than  the  average  score,  matching  instructional  strategies  with  low  and  high  field 
dependent  learners  appears  to  show  some  positive  effect  on  student  learning.  Online  course  instructors  may 
utilize  expository  and  collaborative  types  of  instructional  strategies  for  high  field  dependent  students,  and 
discovery  types  of  instructional  strategies  for  low  field  dependent  students  to  make  the  instruction  more 
appealing  and  effective.  Ultimately  online  students  may  gain  more  learning  benefits  from  the  course  in  terms  of 
their  perceived  learning  outcome,  their  effort  and  involvement,  and  level  of  activity  that  they  perceive  in  the 
online  class. 

The  results  also  revealed  that  there  is  no  single  superior  instructional  strategy  for  high  and  low  field 
dependent  students  among  the  three  types  of  instructional  strategy  used  in  the  study.  The  characteristics  of  low 
field  dependent  students  matched  expository  and  collaborative  instructional  strategies  and  mismatched 
discovery  type  of  instructional  strategies.  When  low  field  dependent  student  groups  were  statistically  compared, 
no  significant  differences  were  detected  for  three  constructs  used  in  the  study.  Matching  and  mismatching 
instructional  strategies  for  low  field  dependent  students  did  not  affect  students’  perceived  learning  outcome, 
their  perceived  effort  and  involvement  in  units,  and  level  of  interactivity  that  they  perceived  during  the  unit. 
Similar  statistical  analysis  was  conducted  for  high  field  dependent  students  whose  characteristics  matched 
discovery  type  of  instructional  strategies  and  mismatched  expository  and  collaborative  type  of  instructional 
strategies.  However,  statistically  no  significant  results  were  found  for  high  field  dependent  students  as  well. 
Results  of  this  study  showed  that  utilizing  expository,  collaborative,  and  discovery  types  of  instructional 
strategies  to  design  online  courses  provided  almost  equal  learning  benefits  for  low  and  high  field  dependent 
students. 

Although,  this  pilot  study  provided  valuable  information  on  gathering  learner  style  information  from 
online  learners,  results  of  the  study  should  be  interpreted  with  caution.  These  findings  may  have  been  due  to  a 
number  of  factors.  Finding  no  significant  results  could  have  been  due  to  small  number  of  subjects.  Considering 
the  fact  that  there  were  twelve-subjects  involved  to  the  study  and  only  three  subjects  were  categorized  as  high- 
field  dependent  individuals,  more  subjects  are  required  to  validate  the  results  of  this  pilot  study.  There  appears 
to  be  other  factors  that  may  have  affected  the  results  of  the  study.  Existing  course  structure  may  not  have 
provided  pure  experiences  in  different  instructional  strategies.  Furthermore,  the  time  allocated  to  complete  units 
was  not  same  so  it  may  have  influenced  the  experiences  of  students  in  three  units.  Finally  the  content  of  units 
were  different  so  the  content  may  have  influenced  the  level  of  effort  that  each  student  put  into  completing  units. 

Future  researchers  should  consider  testing  environments  that  do  strictly  follow  instructional  strategy 
guidelines  to  confirm  these  findings.  Researchers  should  also  consider  testing  other  learning  style  instruments 
and  instructional  strategies  in  their  future  research.  Although  no  significant  differences  were  identified  in  this 
study,  there  is  much  to  learn  about  how  individuals  interact  and  learn  in  online  environments. 
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Introduction 

Many  learning  environments  involve  rituals  for  rehearsal  and  reflection.  Musicians,  for  instance,  spend 
countless  hours  practicing  scales  and  adjusting  their  bodies  to  increase  their  skills.  But  they  do  more  than 
simply  practice:  They  also  play  for  instructors  and  others  who  can  provide  valuable  critiques  of  their 
performances.  Architectural  design  studios  encourage  students  to  create  designs  andshare  them  with  experts 
and  peers  in  organized  “crit”  sessions  that  point  out  good  and  bad  aspects  of  their  work.  Athletic  coaches  often 
watch  videos  of  games  with  their  players  to  reflect  on  issues  for  improvement.  In  all  cases,  there  is  a cycle  of 
skill  rehearsal  followed  by  periods  of  critical  reflection  to  understand  successes  and  failures,  ultimately  to 
improve  future  performance. 

Much  of  reflection  is  about  making  tacit  knowledge  and  routines  explicit  so  they  can  be  analyzed  and 
promote  self-awareness  or  “knowledge-in-action”  (Lin  et  ah,  1999;  Schon,  1983).  In  the  above  examples,  these 
reflections  are  partially  facilitated  through  concrete  artifacts  that  capture  aspects  of  past  performance.  Musical 
sessions  can  be  recorded  to  tape,  architecture  students  create  drawings  and  models,  and  athletes  use  video  when 
reflecting  on  their  skills.  Otherwise  transient  actions  and  performances  are  captured  and  made  explicit  as 
concrete  artifacts  for  reflective  thinking  and  learning. 

Our  research  considers  the  importance  of  making  actions  into  artifacts  for  reflective  thinking.  In 
particular,  we  will  describe  ongoing  efforts  to  develop  computer-based  visualizations  for  diabetes  health 
management.  Approximately  17  million  American  suffer  from  diabetes  (NIDDK,  1998),  and  those  numbers 
continue  to  increase.  The  disease  cannot  be  cured,  but  it  can  be  managed  through  insulin  and  oral  medications 
and  changes  in  diet  and  exercise  habits.  We  are  focused  on  the  latter  part  of  diabetes  self-management,  the 
regulation  of  daily  routines  to  prevent  abnormal  blood  sugar  levels  that  could  lead  to  future  health 
complications. 

Most  diabetics  carry  and  use  glucose  meters,  small,  handheld  devices  that  measure  and  report  current 
blood  sugar  levels.  These  technologies  are  critical  to  diabetic  lifestyles,  as  they  present  physiological  data  to 
help  people  see  how  they  are  dealing  with  the  disease.  Our  research  tries  to  add  additional  information  to 
glucose  meters  by  helping  diabetics  explore  questions  about  why  their  sugar  levels  are  normal  or  abnormal 
during  the  day.  Specifically,  we  developed  a computer-based  visualization  for  displaying  glucose  meter  data 
that  makes  patterns  of  regularity  (or  irregularity)  explicit  to  its  users.  The  hallmark  of  these  visualizations  is  the 
use  of  color  to  provide  global  overviews  of  high,  low,  and  normal  blood  sugars  over  extended  periods  of  time. 

Beyond  visualizing  physiological  data,  we  have  diabetics  take  photographs  of  their  daily  activities, 
focusing  on  things  that  might  impact  their  blood  sugar  levels.  These  images  are  integrated  into  the  computer 
visualizations  to  contextualize  the  numerical  data.  Our  hypothesis  was  that  diabetics  could  begin  to  engage  in 
reflective  thinking  around  their  health  practices  when  provided  with  visualizations  that  point  out  potential 
correlations  between  blood  sugar  levels  (captured  by  glucose  meters)  and  behaviors  (captured  in  photographs). 
We  will  report  results  from  a recent  study  of  the  use  of  visualizations  of  behavioral  and  physiological  data  to 
enhance  the  aspects  of  reflection  stated  in  findings 

Definitions  of  Reflection 

Reflection  has  been  defined  differently  by  different  people.  Before  analyzing  our  study  data  we 
considered  the  following  definitions. 

In  a chapter  in  How  We  Think  (1997),  entitled  “What  is  Thought?”.  Dewey  defines  and  emphasizes  the 
importance  of  reflective  thought.  Reflective  thought  is  one  of  the  four  senses  of  thought:  the  process  of 
accepting  a belief  after  deliberately  seeking  and  examining  its  grounds.  Reflection  involves  the  consequences  of 
ideas  rather  than  merely  a sequence  of  ideas.  Each  created  idea  is  a link  in  a chain  of  ideas.  The  important 
consequences  of  beliefs  or  behaviors  might  force  one  to  consider  the  reasons  for  these  and  come  to  a “ reasoned 
conclusion  Reflective  thought  is  the  “ active , persistent,  and  careful  consideration  of  any  belief  or  supposed 
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form  of  knowledge  in  the  light  of  the  grounds  that  support  it,  and  the  further  conclusions  to  which  it  tends ” 
(Dewey,  p.  6).  Reflective  thinking  has  two  elements  or  subprocesses:  “a  state  of  perplexity,  hesitation,  doubt” 
and  an  investigation  to  confirm  or  refute  the  further  facts  (Dewey,  p.  9).  In  other  words  reflective  thinking 
starts  with  facing  a problem  or  questioning  and  then  continues  with  efforts  at  solving  the  problem  by  reasoning. 
The  process  ends  with  a reasoned  conclusion  which  would  be  the  start  of  another  reflective  thinking  process. 
Dewey  concludes  that  reflective  thinking  “. . . means  judgment  suspended  during  further  inquiry;  and  suspense 
is  likely  to  be  somewhat  painful.”  (1997,  p.  13) 

Schon’s  definition  of  reflection  comes  with  the  following  terms:  knowing-in-action,  reflection-in- 
action  and  reflection  on  action-in-action.  Knowing-in-action  refers  to  the  kind  of  knowing  that  is  revealed  in  our 
intelligent  action  while  executing  a spontaneous  performance  (Schon,  1987).  This  is  a kind  of  action  that  cannot 
be  verbalized.  The  knowing  occurs  in  the  action  as  a response  to  an  unexpected  outcome.  The  unexpected 
outcome  can  be  result  of  anything  that  does  not  happen  as  a part  of  routine.  Further  knowing-in-action  may  help 
us  to  reflect  on  the  unexpected  outcomes  of  daily  routine  and  take  an  in  action.  For  example,  automatically 
steering  a bicycle  to  the  left  in  order  to  maintain  balance  whenever  the  bicycle  tilts  to  the  left  is  an  example  of 
reflection-in  action.  The  action  which  occurs  in  that  present  moment  as  a response  to  the  unexpected  results 
from  and  demonstrates  knowing-in -action.  During  an  action,  we  still  can  make  a change  to  the  situation  and  our 
thinking  serves  to  reshape  what  we  are  doing  while  we  are  doing.  In  cases  like  this,  we  reflect-in-action, 
“thinking  on  our  feet”  (Schon,  1983,  p.  54).  For  example  one  bikes  to  the  left  naturally  when  the  bike  tilts  to  the 
left  at  the  moment  of  in-action.  When  we  think  back  to  see  how  our  knowing-in-action  helped  us  respond  to  the 
unexpected  situation,  then  we  are  reflecting  on  action.  Reflection  on  action  can  help  reveal  our  own  theories  as 
well.  When  one  finds  himself  experiencing  puzzlement,  or  confusion  in  an  uncertain  or  unique  situation,  “he 
reflects  on  the  phenomena  before  him,  and  on  the  prior  understandings  which  have  been  implicit  in  his  behavior 
(Schon,  1983,  p.  68).  This  experience  results  in  generating  “both  a new  understanding  of  the  phenomena  and  a 
change  in  the  situation”  (Schon,  1983,  p.  68). 

Self -reflection  and  reflection  are  the  terms  defined  as  a part  of  self -regulated  learning  (SRL)  by 
Zimmerman  and  Pintrich  (Puustinen  & Pulkkinen,  2001).  Zimmerman  (1998)  defines  self-regulated  learning  as 
a cyclical  process  which  has  three  components:  forethought,  performance  and  self-reflection.  Each  of  these 
components  support  each  other  in  a sequence.  Further  more  specifically  self-reflection  has  four  types  of 
processes:  self-evaluation,  attributions,  self-reactions  and  adaptivity.  As  one  of  the  initial  processes  during  self- 
evaluation,  one  compares  self- monitored  information  with  a goal.  Self-evaluation  leads  one  to  attributions  of 
reasoned  conclusions.  Attributions  lead  one  to  self-reaction  and  also  to  adaptation  to  the  performance. 

Similarly,  Pintrich’ s SRL  definition  has  reflection  as  one  of  the  four  phases,  whereas  the  first  three  are 
forethought,  monitoring,  control  and  reflection  (Puustinen  & Pulkkinen,  2001).  More  “reflection  includes 
cognitive  judgments,  affective  reactions,  making  choices  and  task  and  context  evaluation”. (Puustinen  & 
Pulkkinen,  p.  274)” 


Study  described 

Specifically,  our  study  involved  six  type  I diabetics  and  one  type  II  diabetic.  This  group  took 
photographs  of  their  everyday  activities  for  a month  and  met  with  us  weekly  to  discuss  their  health,  using  their 
visualized  data  as  a conversational  prop  (Brinck  & Gomez,  1992;  Roschelle,  1992)  for  reflection. 

They  were  given  small  digital  cameras  and  asked  to  take  pictures  of  diabetes -related  behaviors.  The 
diabetics  were  free  to  decide  what  they  would  take  pictures  of.  We  suggested  that  they  might  take  pictures  of 
their  meals  and  exercise  activities.  In  their  daily  activities  they  were  testing  and  monitoring  glucose  results  by 
using  a digital  glucose  tester. 

Before  each  weekly  meeting,  we  uploaded  the  glucose  data  and  the  pictures  to  the  computer.  Then  we 
displayed  the  data  with  software  which  was  created  for  this  purpose  (see  Figure  1 and  2).  With  this  interface  we 
could  see  the  glucose  results  and  diabetes -related  activities,  captured  in  pictures,  accompanied  by  a record  of  the 
time  and  day.  For  the  glucose  results,  color  codes  were  used  in  order  to  facilitate  an  easy  and  quick  grasp  of 
glucose  patterns.  The  colors  represented  ranges  of  blood  sugar  levels  as  follows: 

• Dark  blue:  0-39 

• Lighter  blue:  40-80 

• Gray:  81-120 

• Red:  121-140 

• Dark  Red:  141  and  higher 

The  data  for  the  entire  period  were  displayed  and  then  we  posed  questions  to  facilitate  reflection.  When 


20 


necessary,  the  pictures  were  magnified  (see  Figure  3). 

What  follows  is  a discussion,  based  on  preliminary  data  analysis,  of  the  experience  of  the  participants 
and  how  it  helped  them  reflect. 


Findings 

Schon  claims  that  his  examples  of  a baseball  pitcher  who  reflects  on  “winning  habits”  and  a jazz 
musician  who  reflects  on  the  experience  of  making  music,  show  that  “reflection  tends  to  focus  interactively  on 
the  outcomes  of  action,  the  action  itself,  and  the  intuitive  knowing  implicit  in  action”  (1983,  p.  56).  In  our 
study,  participants  were  invited  to  focus  on  the  outcomes  of  activities  related  to  being  diabetic,  activities  by 
themselves  and  the  intuitive  knowing  implicit  in  activity.  Before  taking  photos  of  activities  they  had  to  focus  on 
the  outcomes  of  the  activities  related  to  diabetes.  For  example,  how  does  this  exercise  affect  my  glucose?  Most 
of  them  took  pictures  of  their  meals  and  exercise  activities  such  as  walking  and  weight-lifting.  In  the  process  of 
capturing  activities  the  participant  experiences  the  intuitive  knowing  implicit  in  the  activity.  The  participant 
makes  this  implicit  knowledge  explicit  in  two  ways.  The  participant  makes  the  knowledge  explicit  by 
communicating  via  visual  representations  of  captured  activities,  photos  and  also  by  talking  about  them  later  on 
during  the  interviews.  The  knowledge  is  received  by  two  parties:  the  participant  and  the  interviewer  who  in 
some  cases  could  be  a health  care  person  or  physician. 

In  our  study  we  could  see  how  reflection  occurred  differently  for  different  participants.  When  we 
consider  a consensus  over  given  definitions  of  reflection  above  and  the  outcomes,  we  could  see  four  phases  in 
our  study.  These  phases  were  cyclical,  temporally  overlapping  only  somewhat  linear.  They  corresponded  to  key 
concepts  emerging  from  the  definitions. 

1.  confusion,  perplexity,  hesitation,  doubt,  unexpected  outcome,  judgment  suspended 

2.  judgment  suspended,  investigation,  monitoring,  self-evaluation,  context  evaluation,  reasoning, 
persistent  and  careful  consideration,  knowing-in  action,  reflection-in  action 

3.  reflection-in-action,  reflection  on  reflection-in-action,  new-understandings,  confirm/refute  the 
facts/conclusions,  affective  reactions,  making  choices  and  tasks 

4.  making  choices  and  tasks,  changes 

Following  we  will  introduce  our  preliminary  data  results  associated  with  these  phases. 

Participants’  awareness  about  what  to  do  was  extremely  variable.  Most  of  the  participants  know  what  to  do  in 
general  for  their  health  care.  P108,  who  was  an  athletic,  had  already  come  to  an  understanding  through  past 
experience  of  what  affects  her  blood  sugar  and  how  to  deal  with  it  by  correlating  her  eating  and  exercise.  She 
had  been  exercising  couple  of  times  per  a week  since  her  childhood.  She  kept  log  books  recording  of  what  she 
ate  and  what  she  did  for  exercise.  She  thinks  she  does  not  have  room  to  improve  much.  In  other  words,  she  does 
not  have  much  confusion  in  her  diabetes -related  life.  However  she  still  is  not  sure  about  some  blood  sugar 
changes  that  specifically  occur  due  to  exercise.  She  still  has  not  found  the  reasons  of  this.  This  is  still  a puzzle 
to  her.  When  we  asked  what  she  learned  about  herself  during  the  study,  she  answered: 

“..  .1  don’t  think  I learned  anything  new... because  I am  an  athlete  and  I have  been  at  least  recently 
really  looking  at  my  diabetics  what  affects  it  and  how  to  change  it,  kind  of  understanding  myself  more  what  the 
exercise  does  to  my  health  aham  I think  I have  already  gone  that  process  but  if  maybe  the  other  people  haven’t 
aham  that  they  don’t  really  understand  what  affects  it  as  much  it  might  help  them  for  a better  understanding. 
Since  I wrote  down  everything  I eat  what  insulin  I take,  I can  sort  of  already  look  back  and  see  what  affects 
more. . .1  have  a basic  understanding  but  some  things  are  still  different  like  I said  exercise  will  kick  in  at  strange 
times,  some  times  directly  after  sometimes  later  which  I am  still  trying  to  figure  out. . .for  some  reason  one  day 
my  body  would  kick  in  earlier  or  later”  (P108,  interview4,  conscious,  1:36) 

On  the  other  hand  some  of  the  participants  still  did  not  know  what  causes  what.  They  were  still  in 
confusion  or  hesitation.  And  knowing  the  reasons  for  changes  could  be  helpful  for  P106. 

“...I  hope  to  see  what  it  is  causing  me  to  get  high  blood  sugar... but  if  I see  what  is  causing  me  that  would 
definitely  be  direction...”  (P106,  interviewl,  conscious,  20:03) 

Even  though  they  know,  what  they  do  or  they  do  not  do  to  manage  their  diabetes  has  already  become 
part  of  their  daily  routine.  They  may  not  be  thinking  about  their  diabetics -related  experiences  enough  because 
they  have  become  repetitive  practices  embedded  in  their  daily  routines.  For  example,  Schon  (1983,  p.  61) 
mentions  that  when  practice  becomes  more  repetitive  and  routine,  knowing  becomes  sufficiently  implicit  and 
natural  that  the  practitioner  may  miss  important  opportunities  of  careful  consideration  of  what  he  is  doing.  By 
asking  participants  to  capture  their  diabetes -related  behaviors,  we  invited  them  stand  outside  of  their  routine 
lives  and  think  actively,  persistently  and  carefully  about  what  they  are  doing  for  their  health  care  as  they  are 
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doing  it  (thinking  on  their  feet).  As  a result  of  experiencing  their  lives  out  of  their  routine  by  wearing  different 
lenses  which  conies  with  the  study  requirement,  capturing  behaviors,  they  could  reveal  their  knowing-in-action 
and  then  reflect  in -action. 

For  example  P109  was  not  sure  about  the  reasons  for  his  high  glucose  measures  and  he  was  having 
suspended  judgment: 

“. . .1  think  with  this  really  high  one  I had  a big  dinner  that  evening  that  could  have  been  of  the  causes 
of  that...aham  I know  for  most  of  the  high  ones  they  come  after  meals.  ..that  could  be  something  to  do  with 
eating  a meal  right  before  that  that  could  be  why  they  are  high...”  (P109,  interviewl,  conscious,  24:58) 
and  he  was  reflecting-in-action  by  facing  the  problem  that  needs  to  be  fixed  but  not  having  the  solution/ 
response  yet: 

“. . .1  think  now  looking  at  the  blood  sugars  that  there  are  a lot  of  higher  ones  which  something  needs  to 
be  fixed...”  (P109,  interviewl,  conscious,  27:02) 

While  monitoring  the  two-week  data  on  our  second  interview,  he  continues  reflection-in-action, 
advantageously,  this  time  he  realizes  some  patterns  on  his  high  glucose  results  in  the  mornings  and  questions 
the  reason.  He  wants  to  know  the  reason.  This  is  still  reflection-in-action,  investigation,  since  there  is  an  inquiry 
but  he  does  not  respond  to  the  problem  yet. 

“. . .1  think  I need  to  trying  get  my  blood  sugar  under  control  especially  in  the  mornings.  I have  to 
figure  out  what  is  causing  that.”  (P109,  interview2,  conscious,  56:37) 

P107  started  living  a different  life  when  she  left  home  and  became  a college  student.  This  new  life 
brought  her  some  perplexity  and  suspended  judgment  regarding  her  diabetes  management.  We  can  see  her 
experience  of  four  phases  at  the  same  time:  doubt,  context  evaluation,  making  choices  and  changes. 

“. . . (her  doctor)  said  they  should  have  nutrition  facts  up  there  stuff,  they  do  have  that  for  a lot  stuff  but 
I eat  a lot  at  the  salad  bar  and  they  don’t  have  that  stuff  up  so  unless  I go  and  bug  somebody  in.  I tried  to 
looking  up  on  the  Internet. . .now  just  trying  judging. . .now  I am  trying. . .1  better  go  low  then  high  but  it  is  hard 
to  the  fact  that  at  home  I think  I could  manage  my  diabetes  better... here  my  schedule  I mean  everything  is  so 
different  every  day  is  different  and  I am  walking  everywhere  so  that’s  why  it  is  much  easier  to  control  at 
home...”  (P107,  interview  1,  as  a student-nutrition,  16:29) 

“..  .1  have  been  a lot  more  conscious  about  how  to  cover  my  meals... they  don’t  have  carbs  in  the 
dining  hall.  ..solam  trying  to  guess  and  it  is  the  reason  why  sometimes  it  is  high  or  low. . .so  I have  been  trying 
to  eat  more  regular  salad  dressing  not  fat-free”  (P107,  interviewl,  conscious,  04:17) 

Choosing  the  regular  salad  dressing  shows  her  solution.  She  has  a problem  and  responds  to  that 
problem  while  it  is  occurring.  She  is  reflecting-in-action  and  also  making  choices  and  changes. 

Monitoring  the  glucose  data  and  captured  activities  supported  the  participants  on  their  investigation. 
For  example  both  PI 09  also  PI  10  observe  how  their  eating  amounts  affect  their  blood  sugars  by  monitoring  the 
patterns  on  the  visualized  and  color-coded  data.  They  noticed  that  their  glucose  level  was  high  in  the  mornings 
as  well.  More,  some  of  them  stated  some  new  understandings  of  their  life: 

“..It  was  interesting  to  see  what  kind  of  habits  I have.  Maybe  I realized  what  I should  do  a little  bit 
more..  .Like  I really  never  thought  about  sleeping  how  it  affects  my  sugar...”  (P105,  interview4,  conscious, 

36:27) 

“...this  is  interesting  because  I don’t  usually  view  as  stress  affecting  me  much  but  you  guys  said 
anything  affect  my  blood  sugar  level  that  I could  feel  it. . .like  affecting  me  and  I don’t  usually  give  that  much” 
(P106,  interviewl,  conscious,  10:14) 

“...I  just  became  more  aware  like  at  the  gym.,  .this  is  first  time  I kept  a log  book  for  a long  time  it  kind 
of  helps  seeing  the  pattern.  ..”(P107,  interview3,  conscious,  12:21) 

Monitoring  also  helped  for  realizing  some  habits,  self-evaluation  and  followed  with  confirming  the 

fact: 

“...I  took  a picture  of  glucose  tablets...aham  let  me  see.,  .geez  I don’t  why  I ate  glucose  tablets.  I just 
eat  them  sometimes  when  I feel  low... well  that’s  even  normal  to  deal  with  ...we  (then  he  sees  the  pictures  and 
says)  yeah  I am  actually  starting  to  remember  this  day  I ate  a lot. . .1  ate  what  ever  I want. . .then  I try  to 
compensate  with  extra  insulin  which  is  not  good  idea  I am  trying  to  learn  more  on  that,  me  myself...”  (P106, 
interview3,  conscious,  30:30) 

After  investigation  confirmation  emerged.  Some  participants  noticed  or  confirmed  that  different  kinds 
of  exercising  such  as  walking  versus  weight  lifting  affects  blood  sugar  drops  differently.  For  example  P108 
mentioned  that  weight  lifting  kind  of  activities  affect  later  rather  than  just  immediate  after.  She  already  had  an 
awareness  of  that.  On  the  other  hand  P109  was  not  sure  of  this  and  had  an  idea  about  that  when  questions 
prompted  him  to  see  a correlation: 
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“...maybe  weight  lifting  is  affecting  the  next  day  more  than  the  same  day...”  (P109,  interview4, 
conscious,  45:47) 

“. . .1  think  we  saw  a connection  between  exercise  and  the  next  day  blood  sugar  seems  to  be  lower  in 
the  morning  the  day  after  exercise.  I think  it  is  one  pattern  that  we  saw.  Seems  that  if  I have  a big  dinner  I don’t 
know  how  to  adjust  (?) ”(P  109,  interview4,  conscious,  51:05) 

When  asked  to  provide  the  reasons  for  the  fluctuations  by  monitoring  the  data,  one  participant 
responded, 

“. . . Aham  there  could  be  a lot  of  different  reasons  of  that.  For  instance  if  I am  taking  the  insulin  or  I am 
taking  too  much  insulin  or  eating  I have  noticed  depending  on  what  eat  has  different  effects,  like  if  I eat  pizza  or 
something  the  effects  of  that  don’t  really  impact  me  until  like  two  hours  later.”  (Pill,  interview  3,  measures- 
fluctuations,  36:46) 

In  the  fourth  interview  when  the  color-coded  data  viewed  Pill  realizes  changes: 

“. . .1  see  a significant  improvement  in  the  back  last  five  days. . .” 
and  then  he  comes  to  some  conclusions  when  he  is  asked  to  see  the  connection  between  his  exercise  and  blood 
glucose  data: 

Interviewer  1 : “. . . when  you  look  all  overall  this  data  and  do  you  see  connection  between  your  exercise 
and  blood  glucose  data?” 

Pill:  “definitely” 

Interviewer  1:  “how?” 

“...aham  I think  (?)  last  week  (?)  shows  and  from  when  I work  as  well  shows  that  my  blood  glucoses 
decreases  dramatically  so  if  it  doesn’t  decrease  dramatically  immediately  if  it  doesn’t  immediate  effect  like 
from  running  or  something  aham  symptoms  of  it  will  definitely  show  up  in  the  morning  rather  than  at  night  and 
from  I think  from  work  since  work  is  usually  a longer  period  than  a few  hours  you  see  the  results  of  working  of 
when  I finish  working  my  work  I test  my  blood  and  you  see  the  results...”  (Pill,  interview  4,  reflection-on, 
22:48) 

Ultimately,  some  participants  reported  some  changes  or  considerations  of  changes  in  their  life.  After 
they  responded  to  the  problem  and  then  reflected  on  that.  Some  participants  increased  their  exercise  with  more 
walking  or  decreased  eating  non-recommended  food.  For  example  Pill  stated  that  he  stopped  eating  ice  cream, 
or  PI  10  tried  to  increase  his  exercise  (walking)  and  decided  not  to  eat  late  at  night.  PI  1 1 is  talking  on  the 
experience  of  the  study: 

“...it  was  definitely  a positive  experience  just  because  of  the  fact  that  I just  had  to  be  consciously  aware  of 
yeah  I need  to  take  picture  of  this  and  sometimes  it  is  a horrible  (deter?)  but  sometimes  I think  that  ‘do  I really 
need  to  eat  this?’because  I am  gonna  take  picture  of  it.  I think  it  it  definitely  helped  just  making  me  conscious  of 
what  I eat. . . ” (Pill,  interview4,  DPS-taking  p ictures  -stops  more  eating,  31:19) 

Interviewer  1 follows:  “Do  you  think  it  affected  your  choices  of  eating?” 

“...yeah  a few  times  it  did... like  for  instance  this  whole  past  last  two  months  like  I have  not  eaten  any  ice- 
cream which  is  very  weird  considering  that  I love  ice-cream  and  I am  in  the  Penn  State  (referring  the  popular 
ice-cream  place)...”  (Pill,  interview4,  DPS-taking  pictures -stops  more  eating,  31:19) 

“.. .1  have  improved  the  numbers  (blood  results)... it  is  definitely  worked  a lot...”  (P106,  interview4, 
conscious,  43:31) 

Interviewer  1 asks:  “This  experience  have  helped  your  health?” 

“...this  just  made  me  realize  I am  pretty  out  (?)  of  control  now. ..so  many  little  things  make  big 
differences..  .1  am  trying  to  get  under  control  I guess...”  (P107,  interview4,  conscious,  4:00) 

Limitations  and  Conclusion 

One  of  the  limitations  was  the  interviewers’  lack  of  expertise  on  the  content.  The  interviewers  were  not 
health  experts,  so  that  could  have  decreased  their  ability  to  see  the  correlations  between  the  data  and  the  disease. 
Further,  some  questions  asked  by  the  interviewers  to  facilitate  seeing  the  impact  and  correlation  could  affect  the 
statements  given  by  the  participants.  More  the  tendency  of  pictures  on  the  subjects  of  eating  and  exercise  could 
have  been  the  result  of  examples  given  by  the  researchers  at  the  beginning  of  the  study.  In  addition  to  these,  we 
had  some  technology-related  limitations.  The  camera  was  forgotten  or  inconvenient  to  carry  for  some 
participants.  One  participant  mentioned  that  it  would  be  better  to  have  camera  and  the  monitor  together  in  one 
tool,  since  she  already  has  to  carry  the  monitor.  Also  a few  comments  were  made  about  how  difficult  it  is  to 
communicate  everything  by  taking  pictures.  One  participant  found  writing  log  books  easier  while  some  others 
found  taking  pictures  easier. 

Further  applications  and  research  of  this  study  might  be  in  various  areas  for  various  purposes.  In 
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general,  this  could  be  used  for  reflection  processes  as  a path  to  improvement.  Facilitating  active,  persistent  and 
careful  consideration  of  lived  experiences  by  thinking  about  captured  and  monitored  activities  might  help  one  to 
improve.  We  can  see  applications  in  health  for  complex  disease  diagnosis,  adaptation  and  management.  The 
study  also  has  broader  educational  applications  because  it  describes  and  evaluates  a form  of  self-regulated 
learning. 
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Graphics 


Figure  1 : Glucose  results  on  color-coded  chart 


Figure  2 : Glucose  results  and  pictures  taken 
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Figure  3:  A magnified  view  of  a picture 
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Communities  of  Practice  as  Organizational  Knowledge  Networks 


Stephanie  Allen 
Brigham  Young  University 

Abstract 

As  the  amount  of  critical  information  in  companies  continues  to  burgeon  and  employees’  knowledge  is 
heralded  as  an  organization’s  key  competitive  advantage,  knowledge  management  has  become  a compelling 
workplace  topic  of  discussion.  Communities  of  practice  have  recently  been  recognized  as  effective  means  for 
organizations  to  manage  their  knowledge.  In  order  to  determine  how  virtual  communities  of  practice  serve  as 
knowledge  management  vehicles,  the  authors  of  this  paper  conducted  a study  on  virtual  communities  of  practice 
within  twelve  large,  international  companies  by  interviewing  virtual  community  of  practice  builders  and  leaders 
within  these  organizations.  This  paper  reveals  the  study  ’ s findings.  It  outlines  specific  ways  that  organizations 
can  benefit  from  sponsoring  virtual  communities  of  practice . It  also  identifies  factors  that  community  builders 
and  leaders  can  influence  to  ensure  a VCoP’s  success  as  a knowledge  management  vehicle. 

Introduction 

In  our  current  age  where  information  and  knowledge  are  recognized  as  key  ingredients  for  success  and 
competitive  advantage  (Goldwassar,  2001),  employees’  knowledge,  skills,  and  ideas  are  often  considered  to  be 
companies’  most  valuable  assets  (Schwen,  Kalman,  Hara,  & Kisling,  1998;  Stewart,  1997).  As  the  pressure  to 
manage  knowledge  has  become  more  and  more  pronounced,  companies  are  undertaking  major  initiatives  to 
protect  and  preserve  it  (Wenger,  1998).  Accordingly,  much  literature  has  been  dedicated  to  knowledge 
management  (Blunt,  2001;  Davenport  & Prusak,  1998;  Hamel  & Prahalad,  1994;  Kelly,  1998;  Schwen  et  al., 
1998;  Sharp,  1997;  Pfeffer  & Sutton,  2000;  Wenger  & Snyder,  2000). 

Proponents  of  knowledge  management  primarily  recommend  two  ways  of  capturing  and  managing  an 
organization’s  knowledge  (Hansen,  Nohria,  & Tierney,  2001;  Schwen  et  al.,  1998).  One  recommended  way  to 
manage  knowledge  is  to  codify,  index,  and  warehouse  the  information  (Davenport  & Prusak,  1997;  Davenport 
& Prusak,  1998;  Nonaka  & Takeuchi,  1995;  Schwen  et  al.,  1998).  This  method  is  known  as  the  codification 
means  of  managing  knowledge  (Hansen  et  al.,  2001).  Another  recommended  way  to  manage  knowledge  is  to 
have  the  people  who  generate,  refine,  share,  distribute,  and  use  the  knowledge  actively  manage  it  (Brown  & 
Gray,  1995;  Davenport  & Prusak,  1998;  Pfeffer  & Sutton,  2000;  Wenger,  2000).  This  method  is  known  as  the 
personalization  means  of  managing  knowledge  (Hansen  et  al.,  2001).  A number  of  individuals  who  promote 
actively  managing  knowledge  via  people  recommend  using  informal  workplace  learning  networks,  like 
communities  of  practice  as  knowledge  management  vehicles  (Masterson,  2002;  Pfeffer  & Sutton,  2000). 

This  paper  describes  and  discusses  the  findings  of  a study  that  reveals  specific  ways  that  virtual 
communities  of  practice  can  benefit  individuals  and  organizations  by  helping  them  manage  their  knowledge 
more  efficiently.  To  place  the  research  in  context,  communities  of  practice  are  first  introduced  and  discussed  as 
knowledge  management  vehicles.  An  account  of  the  research  methodology  follows,  along  with  a general 
description  of  the  study  findings.  The  findings  are  then  discussed  in  detail  in  terms  of  gains  that  individuals  and 
organizations  realize  from  participating  in  and  sponsoring  VCoPs.  The  paper  concludes  with  suggestions  of 
how  this  research  contributes  to  the  field. 


Communities  of  Practice  as  Knowledge  Management  Vehicles 

Communities  of  practice  (CoPs)  are  comprised  of  members  who  communicate  one  with  another  to 
generate  and  share  knowledge  and  expertise.  They  function  as  an  interdependent  network  over  an  extended 
period  of  time,  with  the  shared  goal  of  furthering  their  ‘practice’  or  doing  their  work  better  (Wenger,  1998). 
Many  CoPs  operate  virtually  because  community  members  live  and  work  around  the  globe,  relying  on  various 
technological  means  to  communicate  with  one  another.  These  CoPs  are  often  called  virtual  communities  of 
practice  (VCoPs). 

Existing  research  on  communities  of  practice  indicates  that  they  have  been  found  to  help  employees 
acculturate  themselves  into  an  organization  (Chao,  2001;  Gregory,  1993)  develop  a work -associated  identity 
(Hara,  2000;  Yi,  1999),  teach  them  skills  to  get  their  work  done  more  efficiently  (Brown  & Gray,  1995;  Sharp, 
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1997),  motivate  them  to  do  their  work  (Barab  & Duffy,  2000;  Bradsher  & Hagan,  1995),  and  improve  their 
individual  job  performance  (Allen,  2003).  However,  literature  relating  the  individual  and  organizational  impact 
of  CoPs  to  knowledge  management  is  limited. 

Wenger  and  Snyder  (2000)  reported  that  CoPs  have  helped  several  organizations  improve  their  overall 
performance,  enhance  their  communication  structure,  and  support  their  goals.  However,  beyond  a few 
individual  case  studies  and  Wenger  and  Snyder’s  case  study  synthesis,  little  research  focuses  on  the  ways  that 
CoPs  facilitate  knowledge  management  within  organizations.  In  order  to  learn  more  about  the  ways  that  CoPs, 
and  particularly  VCoPs,  help  organizations  and  individual  VCoP  members  manage  their  knowledge,  a study 
was  conducted  on  VCoPs  within  large  corporations,  government  organizations,  and  educational  institutions. 

Research  Methodology 

The  research  team  utilized  interviews  as  the  primary  research  data  source  because  they  are  a key  means 
of  gathering  qualitative  case  study  data  (Stake,  1995;  Yin,  1994).  They  used  a survey  as  a secondary  research 
data  source.  The  researchers  collected  the  study  data  in  two  phases.  They  interviewed  VCoP  builders  and 
leaders  in  phase  one.  During  phase  two,  the  researchers  collected  data  from  VCoP  participants  via  a web-based 
survey.  Prior  to  starting  each  phase,  the  researchers  pilot  tested  and  validated  the  data  collection  instruments 
with  a small  group  of  selected  potential  participants  to  ensure  that  the  collected  data  would  be  reliable  and  valid. 

The  research  participants  involved  in  phase  one  consisted  of  twenty-five  builders  and  leaders  of  virtual 
communities  of  practice  from  thirteen  organizations.  These  organizations  were  primarily  large,  global 
companies  who  provide  a wide  range  of  products  and  services,  such  as  financial  services,  microprocessor  chips, 
technology  services,  training,  health  care  services,  consumer  products,  and  insurance.  The  interviewees  were 
carefully  screened  before  they  were  interviewed  to  ensure  that  the  virtual  communities  of  practice  they  lead  or 
built  adhered  to  the  theory  and  practices  of  community  of  practices.  Once  the  VCoPs’  legitimacy  was 
established,  formal  interviews  were  conducted  via  60-90  minute  telephone  conversations.  These  interviews 
were  recorded  and  later  transcribed 

The  interviews  addressed  all  aspects  of  VCoPs  in  order  to  gain  a holistic  understanding  of  their 
utilization.  Some  interview  questions  focused  around  discovering  why  VCoPs  exist  within  organizations,  what 
kind  of  “work”  they  do,  and  how  they  get  started.  Other  questions  attempted  to  ascertain  how  VCoPs  impact  the 
flow  of  data,  information,  and  knowledge  throughout  their  organizations.  For  example,  interviewees  were  asked 
to  identify  and  explain  how  the  communities  of  practice  served  as  forums  to  create  and  disseminate  knowledge 
throughout  their  organizations. 

During  the  second  phase,  the  research  team  used  the  data  gathered  in  phase  one  to  create  a web-based 
survey  that  was  distributed  via  e-mail  to  participants  of  VCoPs  in  forty  organizations.  Approximately  150 
virtual  community  of  practice  members  responded  to  the  web-based  survey.  The  survey  respondents  worked  for 
financial  institutions,  microprocessor  chip  manufacturers,  technology  services  companies,  training  providers, 
health  care  providers,  consumer  product  developers  and  distributors,  insurance  providers,  and  various  other 
corporations.  The  survey  questions  focused  around  discovering  why  participants  join  and  participate  in  VCoPs, 
what  factors  contribute  to  a VCoP’s  success  or  failure,  what  learning  medium  participants  prefer  for  various 
types  of  learning  and  information  exchange,  and  how  learning  in  VCoPs  compares  to  learning  via  more  formal 
training  methods.  The  web-based  survey  software  automatically  captured  the  participants’  responses. 

Findings 

In  general,  the  findings  indicate  that  individuals  and  organizations  can  receive  substantial  knowledge 
management-related  benefits  from  participating  in  and  sponsoring  VCoPs.  The  interviews  with  VCoP  builders 
and  leaders  and  survey  responses  uncovered  a wide  range  of  explicit  and  tacit  knowledge  about  virtual 
communities  of  practice.  The  interviewees  articulated  how  the  VCoPs  actually  perform  in  applied  contexts  and 
explained  the  benefits  they  provide  to  their  members  and  to  the  organizations  that  sponsor  them.  The  survey 
data  revealed  ways  that  members  believe  VCoPs  can  significantly  help  individuals  learn,  manage  their 
knowledge,  and  perform  their  jobs  better. 

Ways  that  VCoPs  Benefit  Individual  Members 

During  the  course  of  the  study,  the  data  provided  both  by  VCoP  builders  and  participants  suggest  that 
VCoPs  benefit  members  by  helping  them  do  their  jobs  better,  support  their  learning  process,  and  extend  or  share 
their  knowledge  with  one  another.  In  their  interview  responses,  builders  of  various  virtual  communities 
explained  how  VCoPs  support  their  members’  learning  processes.  For  example,  one  VCoP  builder  stated; 
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[VCoPs]  form  because  that’s  how  people  actually  learn.  That’s  how  they  share  information,  share 
resources...  ask  questions,  get  answers  to  those  questions,  and  so  forth.  This  is  how  people  really  do 
work  and  we  in  the  training  world,  since  that  is  where  1’nr  coming  from,  we  often  look  from  an 
educational  training  perspective  that  most  learning  goes  on  in  the  classroom,  but  the  reality  is 
obviously  that  it  doesn’t.  Most  of  it  goes  on  in  the  workplace. . . most  of  the  real  critical  learning  goes 
on  in  the  workplace. 

In  general,  the  study  data  suggest  that  VCoPs  help  employees  learn  by  situating  learning  in  the  workplace, 
providing  just-in-time  and  context-specific  solutions  to  problems,  and  increasing  employee  interaction. 

VCoPs  Situate  Learning  in  the  Workplace 

Recent  research  shows  that  employees  often  learn  more  in  the  workplace  than  they  do  in  formal 
training  environments  because  they  fail  to  transfer  and  implement  formal  training  into  their  jobs  (Gilbert,  1978; 
Mager,  1992;  Sorohan,  1993;  Stolovitch  & Keeps,  1999).  VCoPs  are  valuable  on-the-job  learning  environments 
because  they  are  situated  in  employees’  immediate  work  setting.  One  VCoP  builder  stated  that  “a  community  of 
practice  is  a better  vessel  even  to  do  training  [than  formal  training  environments]  because  it’s  done  within  the 
situation  of  the  work  [they’re]  doing.” 

VCoPs  Provide  Just-in-time  Solutions  to  Problems 

VCoPs  also  provide  a context  where  individual  members  gain  access  to  people  and  resources  that  can 
help  them  solve  their  problems  by  viewing  their  problem  from  multiple  perspectives  and  generating  numerous 
problem-solving  ideas.  In  other  words,  VCoPs  provide  a way  for  members  to  discover  solutions  to  problems 
when  they  are  most  needed- just  in  time  and  in  their  own  unique  context.  This  significantly  increases 
employees’  performance  capabilities. 

As  part  of  the  web-based  survey,  VCoP  members  were  asked  to  identify  the  benefits  they  personally 
gain  from  participating  in  virtual  communities.  In  response  to  this  question,  99%  of  participant  responded  that 
‘‘job  skills  and  knowledge”  were  the  most  important  benefits.  Approximately  84%  of  participants  also  indicated 
that  VCoPs  provide  ‘‘excellent  problem  solving  resources.”  The  percentages  of  these  responses  clearly  suggest 
that  people  join  and  participate  in  VCoPs  to  ‘‘gain  knowledge  and  skills”  and  to  ‘‘access  resources”  that  will 
empower  them  to  make  better  decisions. 

VCoPs  Increase  Employee  Interaction 

VCoPs  also  benefit  individual  members  because  the  group  structure  of  VCoPs  allows  members  to 
share  information  and  engage  in  learning  activities  with  peers.  The  increased  peer  interaction  then  leads  to  an 
increase  in  the  members’  knowledge  retention  and  stronger  relationships  across  the  organization.  This  is 
particularly  true  when  VCoPs  involve  people  from  across  the  organization,  around  the  world,  and  different  job 
descriptions  and  specializations.  Because  of  these  variations,  the  input  and  connections  made  by  VCoP 
members  unite  employees  and  help  them  gain  a sense  of  purpose  and  awareness  that  their  individual  efforts 
contribute  to  organization-wide  strategies.  In  this  way,  community  participation  and  interaction  reduces  both 
hierarchical  and  geographical  boundaries  and  increases  employee  unification. 

In  a related  survey  question,  VCoP  participants  selected  their  top  three  reasons  for  participating  in 
virtual  communities.  Participants  selected  their  reasons  from  a variety  of  responses  related  to  relationship 
building,  productivity,  status,  and  motives.  In  response  to  this  question,  the  participants  indicated  that 
developing  ‘‘professional  relationships  with  other  community  members”  is  their  top  reason  for  participating  in 
VCoPs.  This  response  specifically  relates  to  knowledge  management  because  developing  professional 
relationships  with  other  VCoP  members  is  a key  step  in  transferring  knowledge  among  members.  Also,  when 
members  exchange  information  and  aid  one  another  in  solving  problems,  the  overall  productivity  of  the 
community  increases.  This  increase  in  job-performance  implies  that  employees  are  learning  how  to  do  their  jobs 
better. 

The  benefits  of  situating  learning  in  the  workplace,  providing  just-in-time  and  context-specific 
solutions  to  problems,  and  increasing  employee  interaction  were  reinforced  by  a final  question  from  the  web- 
based  survey,  which  asked  VCoP  members  to  identify  whether  VCoPs,  web-based/computer-based  training, 
instructor-led  training,  or  mentor/ apprentice  learning  environments  best  help  them  accomplish  certain  aspects  of 
their  jobs.  In  response  to  this  survey  question,  VCoP  members  indicated  that  participating  in  VCoPs  helps  them 
do  the  following  six  specified  job  aspects  better: 

• Providing  for  an  efficient  idea  exchange 

• Generating  a broad  perspective  on  solving  problems 
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• Providing  greater  access  to  experts 

• Increasing  members’  knowledge 

• Increasing  members’  motivation  to  learn 

• Effectively  helping  others  learn 

Additionally,  VCoPs  ranked  second  in  conjunction  with  “providing  a more  direct  solution  to  a problem’’  and 
“impacting  people’s  attitude  about  their  jobs.’’  These  responses  suggest  that  VCoP  members  view  VCoPs  as  a 
learning  environment  that  helps  them  manage  their  knowledge  and  better  perform  on  the  job. 

Ways  that  VCoPs  Benefit  Organizations 

Organizations  also  gain  key  knowledge  management-related  benefits  from  supporting  virtual 
communities  of  practice.  As  previously  indicated,  knowledge  management  initiatives  revolve  around  moving 
data,  information,  and  knowledge  effectively  throughout  an  organization.  The  general  goal  is  to  decrease  the 
communication  barriers  that  exist  in  nearly  every  organization  between  individuals  due  to  divisions,  levels 
within  the  organization,  and  physical  locations.  In  their  interviews,  all  of  the  virtual  community  of  practice 
builders  and  leaders  stated  that  VCoPs  improve  their  organization’s  knowledge  management  initiatives  via 
either  direct  or  indirect  means.  They  indicated  that  virtual  communities  of  practice  facilitate  a greater  flow  of 
information  across  organizations  by  breaking  down  many  of  the  existing  barriers.  They  also  indicated  that 
VCoPs  increase  the  networking  and  communication  opportunities  available  to  VCoP  members  across 
organizations  by  providing  increased  interaction  between  organizational  units  where  communication  was 
previously  impossible,  increasing  exchanges  between  management  and  employees,  extending  discussions  that 
occur  in  face-to-face  meetings,  and  creating  a written  repository  of  best  practices  that  VCoP  members  have 
ongoing  access  to.  VCoPs  also  appear  to  increase  the  informal  training  that  occurs  within  the  organization, 
foster  innovation,  and  instigate  cost  savings.  Each  of  these  benefits  is  elaborated  upon  below. 

VCoPs  Increase  Interaction  Among  the  Best  Minds 

Before  VCoPs  existed,  employees  that  spanned  geographic  borders  or  time  zones  were  limited  in  their 
ability  to  effectively  share  data,  information,  and  knowledge  with  one  another.  Virtual  communities  of  practice 
overcome  time  and  physical  boundary  limitations.  Thus,  they  grant  community  members  access  to  the  best  and 
brightest  human  resources  throughout  an  organization,  no  matter  what  business  unit  or  country  they  reside  in. 
One  community  builder  explained  this  benefit  when  he  stated,  “a  virtual  community  of  practice  expands  the 
quality  of  the  skill  base  that  we’re  able  to  draw  from.. . it  helps  us  get  the  best  quality  people ...  by  not  being 
constrained  by  physical  location.” 

By  granting  open  access  to  the  knowledge  and  expertise  of  a collective  whole,  VCoPs  increase  their 
members’  power  and  ability  to  function  effectively  and  efficiently.  For  example,  VCoP  members  often  help 
individuals  solve  problems  that  they  couldn’t  solve  on  their  own  in  a relatively  short  period  of  time.  By  posing  a 
problem  to  VCoP  members  around  the  globe,  these  individuals  can  get  a variety  of  contextualized  solutions  to 
their  exact  problem  or  recommendations  based  on  similar  problems  that  other  VCoP  participants  have 
experienced  very  quickly.  One  VCoP  builder  emphasized  the  problem-solving  benefit  of  VCoPs  in  the 
following  statement: 

There  is  no  time  barrier,  no  geographic  barrier,  no  culture  barrier  keeping  you  from  solving  your 
problem. . . . If  you  are  in  the  middle  of  Kalimantan,  or  in  the  middle  of  the  Borneo  Islands,  you  have  the 
company’s  [virtual  community  members]  and  with  that,  you  have  the  support  of  everywhere  in  the 
world  helping  you  resolve  your  problem. 

VCoPs  Increase  Communication  between  Employees  and  Management 

Another  way  that  VCoPs  boost  an  organization’s  data,  information,  and  knowledge  flow  is  increasing 
the  communication  between  management  and  employees.  VCoPs  do  this  by  providing  a means  for  employees  to 
safely  voice  opinions  and  concerns  and  introducing  a channel  through  which  management  can  solicit  individual 
employee  input  and  feedback.  The  exchange  both  directions  helps  to  break  down  the  hierarchies  that  separate 
management  from  front  line  employees.  For  example,  many  community  builders  indicated  that  individual  VCoP 
members  who  hesitate  to  voice  their  opinion,  share  their  ideas,  and  help  create  new  processes  on  their  own  will 
engage  in  these  activities  within  their  virtual  communities.  The  results  of  these  exchanges  can  be  shared  directly 
with  members  of  the  company’s  management  if  they  are  VCoP  members.  If  not,  the  VCoP  members  are  able  to 
discuss  the  issue(s)  raised  among  themselves,  and  then  pass  their  collective,  and  usually  improved  opinions, 
ideas,  and  recommendations  on  to  company  management  from  the  collective  community  membership  instead  of 


30 


a single  individual.  In  that  situation,  VCoP  members’  feedback  and  recommendations  often  have  more  credence 
with  management  because  it  is  collective,  rather  than  feedback  from  a single  individual. 

In  the  same  manner  that  VCoPs  provide  members  with  a safe  vehicle  to  communicate  with 
management,  VCoPs  provide  managers  with  a non-invasive  mechanism  to  inform  employees  of  new 
developments,  influence  policy  and  solicit  feedback.  In  her  interview,  one  VCoP  builder  indicated  that  her 
management  actively  uses  VCoPs  to  gather  input  from  employees  in  remote  locations,  especially  when  they  are 
building  new  programs  that  will  impact  those  employees.  She  said  that  the  VCoPs  provide  an  easy  way  to  get 
feedback  on  new  programs  or  policies  from  people  across  their  organization  instead  of  only  receiving  input 
from  employees  in  the  corporate  office  as  they  did  in  the  past.  In  addition  to  facilitating  the  creation  of  better 
policies  and  programs,  VCoPs  inadvertently  increase  employee  buy-because  they  have  opportunities  to  voice 
their  opinion  about  those  policies  and  programs . Additionally,  virtual  communities  of  practice  establish  a forum 
for  company  management  and  employees  to  collectively  establish  standards  across  geographic  and  culture 
boundaries. 

VCoPs  Extend  Communication  Between  Face-to-Face  Meetings 

VCoPs  also  increase  the  information  flow  between  employees  by  extending  communication  that  occurs 
in  face-to-face  meetings.  Members  of  VCoPs  who  meet  physically  on  a regular  basis  often  use  the  virtual 
component  of  their  communities  to  interact  with  one  another  between  these  meetings.  As  an  example,  one 
VCoP  leader  said  that  the  virtual  communication  channel  between  members  of  his  virtual  community  allows 
them  to  follow  up  on  issues  and  stay  abreast  of  concerns  facing  their  organization,  divisions,  or  product  lines 
when  they  are  not  physically  present.  It  also  gives  community  members  the  chance  to  jointly  work  on 
unresolved  issues  and  action  items  assigned  during  face-to-face  meetings  before  the  next  meeting  takes  place. 
For  example,  one  VCoP  builder  commented  that  the  addition  of  a virtual  component  to  a co-located  community 
of  practice  greatly  increased  communication  between  community  members  because  they  were  able  to  discuss 
items  and  follow  up  on  tasks  that  they  had  previously  forgotten  between  face-to-face  meetings.  This  made  both 
the  face-to-face  meetings  and  the  period  of  time  between  those  meetings  more  productive. 

VCoPs  Codify  Best  Practices  and  Solutions  to  Problems 

Another  positive  VCoP  outcome  related  to  knowledge  management  is  the  manner  in  which  the 
information  is  captured.  Since  most  of  the  communication  and  information  exchange  originating  in  virtual 
communities  of  practice  occurs  electronically,  the  questions,  solutions,  and  best  practices  exchanged  can  be 
captured,  organized,  and  archived  for  reference  at  a later  date,  in  addition  to  being  shared  among  community 
members  immediately.  This  practice  positively  impacts  an  organization’s  knowledge  management  initiative 
from  both  the  codification  and  personalization  knowledge  management  perspectives  introduced  at  the  beginning 
of  this  paper.  Through  VCoPs,  information  is  shared  between  people  in  relation  to  a specific  context;  thus  it  is 
personalized.  Additionally,  because  it  is  digitally  captured,  it  is  also  codified  to  that  others  can  benefit  from  the 
exchange  in  the  future. 

One  example  of  such  a beneficial  exchange  occurred  in  a VCoP  hosted  by  a multi-national  insurance 
provider.  In  this  instance  when  individuals  in  a Canadian  office  said,  ‘‘Boy  we’ve  got  a problem  with...” 
members  of  the  VCoP  were  able  to  say,  ‘‘Oh,  the  Midwest  office  has  already  figured  it  out  and  they’ve  already 
got  the  solution  implemented.  Talk  to  them  and  look  at  the  information  in  the  community  archives  to  see  what 
they  did.”  In  that  single  instance,  the  VCoP  provided  both  the  personal  contacts  and  the  repository  of 
information  that  saved  the  Canadian  office  countless  hours  of  duplicate  effort  and  a large  amount  of  money. 

VCoPs  Facilitate  Informal  and  Formal  Training 

Due  to  the  nature  of  the  communication  that  occurs  in  VCoPs,  these  communities  regularly  facilitate 
informal  training.  In  many  ways,  the  questions  that  VCoP  participants  pose  to  one  another  and  their  calls  for 
problem  solving  assistance  serve  as  informal  training  requests  because  one  VCoP  member  is  seeking 
information  from  peers  or  experts  to  perform  his  or  her  job  better.  If  community  interaction  is  viewed  in  this 
light,  whenever  VCoP  members  answer  one  another’s  questions  or  provide  advice,  they  are  filling  those 
informal  training  requests  with  impromptu  training  experiences.  As  a result,  virtual  communities  of  practice  act 
as  informal  training  networks  and  they  provide  constant  informal  training  and  mentoring  opportunities.  VCoP 
leaders  indicated  that  community  members  also  impact  and  influence  formal  training  opportunities  by 
identifying  where  collective  knowledge  gaps  exist  and  requesting  formal  training  for  VCoP  members  in  order  to 
fill  those  gaps . 
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VCoPs  Foster  Innovation 

Virtual  communities  of  practice  also  benefit  their  sponsoring  organizations  by  fostering  innovation. 
VCoP  builders  and  leaders  indicate  that  VCoPs  tend  to  foster  innovation  and  refine  organizational  processes 
because  the  me  rnbers  examine  problems  and  processes  from  multiple  perspectives  in  a non-threatening,  non- 
hierarchical  non-constrained  environment  and  often  create  new  processes  or  streamline  existing  processes  . One 
VCoP  expert  focused  in  on  this  organizational  benefit  in  her  interview  when  she  said,  “If  you  bring  the  right 
people  together  and  help  them  share  their  tasks  and  knowledge  and  collaborate  on  problem  solving,  you  have 
innovation. . . so  a community  of  practice  is  a perfect  structure  to  engender  innovation.”  Other  VCoP  builders 
echoed  this  expert’s  assertion  when  they  told  us  that  their  VCoP  members  generally  get  excited  about  having 
opportunities  to  innovate  and  take  advantage  of  those  opportunities . 

VCoPs  Help  Organizations  Reduce  Costs 

Virtual  communities  of  practice  can  help  companies  reduce  their  bottom  line  in  a variety  of  ways. 
VCoP  builders  and  leaders  indicated  that  by  sponsoring  virtual  communities  of  practice  they  were  able  to 
reduce  the  need  for  travel,  decrease  their  overall  training  budget,  reduce  duplicated  efforts,  and  minimize  the 
time  it  takes  to  communicate  with  VCoP  members  across  the  world.  In  particular,  several  community  builders 
noted  that  by  utilizing  virtual  communities  of  practice  as  a training  and  collaboration  tool,  they  have  been  able 
to  significantly  decrease  the  overall  training  expenses  for  their  organization.  For  example,  one  VCoP  builder  of 
a worldwide  oil  product  company  indicated  that  his  organization  saves  $35,000 -$40,000  every  month  because 
members  of  a specific  community  of  practice  discuss  and  work  on  issues  that  they  used  to  discuss  in  a monthly 
face-to-face  meeting  through  the  VCoP.  This  builder  said  that  even  though  VCoPs  require  specific  software  and 
other  technology  to  communicate,  the  costs  associated  with  that  technology  are  substantially  less  than  paying 
for  all  the  community  members  to  meet  physically  on  a regular  basis  so  he  happily  provides  the  needed 
software,  hardware,  and  server  space  needed  by  the  VCoP  to  operate  effectively. 

Several  community  builders  also  noted  that  sponsoring  virtual  communities  can  help  organizations 
reduce  costs  incurred  by  duplicating  efforts.  One  VCoP  builder  stated  that  VCoPs  “help  out  the  bottom  line”  as 
they  start  to  reduce  redundancies  and  eliminate  duplicated  efforts.  Another  VCoP  leader  quantified  the  costs 
savings  achieved  by  a particular  community  in  terms  of  his  personal  time  savings.  He  commented  that  he  is  able 
to  achieve  the  same  gains  by  relaying  a message  to  members  of  this  community  virtually  in  a couple  of  hours  as 
he  did  when  he  traveled  for  two  or  three  days  to  relay  the  same  message  with  community  members  physically . 

Conclusion 

The  research  data  indicate  that  virtual  communities  of  practice  benefit  the  knowledge  management 
initiatives  of  the  organizations  that  sponsor  them.  The  interviewees’  comments  evidence  the  fact  that  the 
interaction  between  VCoP  members  increases  the  amount  of  data,  information,  and  knowledge  that  is 
exchanged  throughout  an  organization  and  heightens  employees’  awareness  of  what  others  are  doing.  The 
information  exchange  mutually  benefits  community  members  throughout  organizations  and  the  organizations 
themselves.  The  data  also  indicate  that  VCoPs  increase  opportunities  for  training  and  innovation,  and  help  to 
reduce  costs.  These  combined  benefits  lead  to  improved  management  of  an  organization’s  knowledge  because 
they  facilitate  knowledge  generation,  knowledge  codification  and  coordination,  and  knowledge  transfer 
throughout  organizations  (Davenport  & Prusak,  2000). 

This  study’s  findings  contribute  to  the  growing  body  of  knowledge  about  virtual  communities  of 
practice  by  identifying  specific  ways  that  organizations  benefit  from  sponsoring  VCoPs  and  by  revealing  critical 
factors  that  organizations  can  influence  to  strengthen  the  likelihood  that  these  communities  will  succeed.  As  a 
result,  the  data  are  valuable  for  organizations  who  desire  to  strengthen  their  ability  to  manage  their  collective 
employees’  knowledge  and  utilize  virtual  communities  of  practice  in  that  endeavor.  The  VCoP  builders  and 
leaders  who  participated  in  the  study  collectively  articulated  that  greater  information  and  communication  flow, 
increased  opportunities  for  training  and  innovation,  better  utilization  of  global  resources,  and  reduced  costs  are 
benefits  that  organizations  can  realize  from  supporting  virtual  communities  of  practice.  These  benefits  directly 
lead  to  improved  knowledge  management  on  both  the  individual  and  organizational  level  because  the  virtual 
communities  of  practice  serve  as  organizational  networks  where  knowledge  is  created,  disseminated,  and 
transferred  throughout  organizations . 
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Introduction 

The  existence  of  a "digital  divide”  in  which  portions  of  society  do  not  have  sufficient  access  to 
technology,  nor  to  the  information  and  skills  that  technology  use  imparts,  was  of  concern  for  educators  and 
policy  makers  even  before  home  computers  were  easily  connected  to  the  Internet  (NTIA  1995,  1998,  1999, 

2000;  Resmer,  Mingle  & Oblinger,  1995;  Riley,  1996;  Annie  E.  Casey  Foundation,  1998).  As  new  technologies 
(such  as  broadband  Internet  access),  and  new  uses  for  technology  (such  as  interactive  websites  for  homework) 
are  developed,  new  levels  of  the  divide  will  appear,  certainly  to  the  extent  that  children  in  low  income  homes  do 
not  gain  access  to  the  same  level  of  information  at  the  same  rate  as  other  children  (c.f.  Hundt,  2003;  Digital 
Divide  Network,  2003).  In  an  era  in  which  funding  may  be  difficult  to  obtain,  the  development  of  tested, 
scalable,  affordable  solutions  should  therefore  be  a mandate  for  educators. 

Research  in  the  digital  divide  field  is  still  in  the  critical  early  stages;  consequently,  much  of  the 
research  has  focused  on  demographic  information  and  on  documentation  of  participant  benefits  (cf.  Chow,  Ellis, 
Walker  & Wise,  2000).  As  it  is  now  well  established  that  participants  do  benefit,  and  that  those  who  would  not 
otherwise  have  access  to  technology  do  take  advantage  of  such  opportunities,  more  focused  research  is  called 
for.  In  addition,  much  early  digital  divide  research  has  been  based  on  solutions  developed  in  conjunction  with 
community  technology  centers  (Chow,  Ellis,  Walker  & Wise,  2000);  the  notable  exceptions  (e.g.  Apple 
Classroom  of  Tomorrow,  1995;  Pinkett,  2000,  2002)  were  largely  based  on  partnerships  with  specific 
institutions  and  on  highly  funded  pilot  programs,  so  that  conclusions  drawn  may  be  limited  in  the  extent  to 
which  they  can  be  used  to  plan  larger  programs  and  fiscally  reasonable  solutions.  Moreover,  as  Secretary  of 
Education  Rod  Paige  has  pointed  out  (US  Dept,  of  Ed.,  2003a,  2003b,  2003c),  access  in  schools  and  in  other 
institutions  such  as  libraries  does  not  substitute  for  access  to  home  computers  with  Internet  connectivity. 

Projects  designed  to  test  home -based  technology  programs  are  thus  seen  to  be  a priority  (Andrews,  DiGangi,  & 
Jannasch-Pennell,  1999,  2000;  Stock,  2001). 

This  paper  reports  on  a seven-year  project  that  was  specifically  designed  to  generate  scalable  and 
affordable  solutions  to  the  digital  divide  in  a participatory  research  setting,  and  on  a briefer  study  based  on  the 
project;  one  year  was  allocated  to  data  collection  in  the  study,  and  a second  year  was  devoted  to  completing  the 
analysis.  The  paper  offers  an  overview  of  results  of  the  study,  and  of  the  means  by  which  the  results  were 
obtained.  While  more  detailed  papers  are  planned,  it  is  hoped  that  this  overview  will  draw  attention  not  only  to 
the  divide,  but  also  to  some  solutions,  and  to  the  efficacy  of  a grassroots  -oriented  means  of  generating  such 
solutions. 

In  the  seven-year  project,  computers  were  placed  in  low  income  homes  as  well  as  in  non-standard  dwelling 
places  (e.g.  shelters  and  vehicles)  via  a participant-led  grassroots  technology  program,  Floaters.org.  Outcomes 
regarding  disposition  and  use  of  the  computers  were  tracked  largely  via  a peer  mentoring  process  in  which 
previous  recipients  introduced  new  members  to  the  computers.  Mentors  in  particular,  and  mentees  as  well,  were 
invited  to  bring  their  experiences  and  suggestions  to  a participant  research  group  and  in  so  doing  to  serve  as  a 
bridge  to  trained  researchers.  The  participant  research  group  met  weekly  and  functioned  as  a consensus -based 
decision-making  body  with  regard  to  the  evolution  of  the  project.  Participants  thus  shaped  the  program  to  their 
needs  in  an  exemplary  participatory  action  research  project  (c.f.  Moser,  1977;  Fals  Borda,  1995;  Sohng,  1996). 

Five  years  into  the  project,  a mixed-method  study  was  begun  that  would  focus  on  specific  aspects  of 
home-based  technology  integration.  Methods  were  chosen,  with  participant  input,  to  maximize  the  amount  of 
information  gained.  Thus,  discourse  analysis  was  selected  as  the  appropriate  method  to  examine  transcripts  of 
taped  peer  mentoring  sessions;  grounded  theory  was  used  to  identify  changes  to  the  program  that  emerged  as 
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the  program  was  scaled  up  in  size;  and  a standard  self-efficacy  scale  was  used  to  examine  participant  self 
efficacy  with  regard  to  technology  occupations.  Finally,  the  program  looked  back  at  itself  via  the  process  of 
participatory  action  research,  used  in  the  project  in  order  to  generate  and  evaluate  digital  divide  solutions  from 
the  bottom  up,  that  is,  with  maximum  input  from  the  people  most  affected. 

Purposes  of  the  Project  and  Study,  and 
Research  Questions  for  the  Study 

Participant -chosen  goals  for  the  project,  and  therefore  for  the  participatory  aspect  of  the  study  as  well, 
were  to  investigate  how  best  technology  could  be  integrated  into  the  homes  and  lives  of  low  income 
populations;  and,  consequently,  how  technology  could  be  used  to  improve  these  lives.  From  the  beginning,  the 
project  sought  not  only  demographic  information  and  information  about  participant  benefits,  but  also 
information  about  how  those  benefits  could  be  accomplished. 

Goals  of  the  study,  as  contrasted  with  those  of  the  project,  were  chosen  in  participant-driven  focus 
groups  in  which  members  learned  research  methods.  These  goals  focused  specifically  on  a set  of  particular 
research  questions: 

What  took  place  with  regard  to  teaching  and  learning  in  the  peer  mentoring  sessions? 

How  did  the  structure  of  the  program,  designed  over  time  by  participants,  change  further  as  the 
program  was  scaled  up  in  size? 

Did  participant  self  efficacy  change  during  the  study  with  regard  to  occupations,  especially  with  regard 
to  technology  occupations? 


About  the  Floaters.org  Project 

Working  with  university  researchers,  a community  group  designed  and  implemented  the  Floaters.org 
project,  in  which  older  computers  were  and  are  recycled  into  low-income  homes  and  then  monitored.  The 
Floaters.org  project  itself  is  a participatory  action  research  project,  wherein  all  participants  have  an  equal  voice 
and  goals  are  set  by  group  members  working  together.  The  project  was  designed  to  integrate  technology  with 
those  who  are  least  likely  to  otherwise  attain  it:  specifically,  those  living  in  poverty,  those  who  have  been 
homeless  and  later,  people  with  disabilities  who  could  not  otherwise  afford  technology.  The  project 
simultaneously  was  designed  to  identify  and  put  into  practice  the  highest  pedagogical  and  research  practices. 
This  does  not  imply  that  mistakes  were  not  made,  but  rather  that  learning  from  mistakes  was  a built-in  part  of 
the  process. 

An  examination  of  the  nature  of  the  project,  undertaken  as  the  two  year  study  began,  showed  that 
participants  took  on  three  roles:  peer  mentors  or  mentees;  shapers  of  the  program;  and  co-researchers.  Each  role 
corresponds  to  one  of  the  research  questions  stated  above. 

The  monitored  study  was  designed  with  input  from  participants,  and  participant  approval  was  gained  for  all 
aspects  of  the  study.  As  part  of  their  work  as  co-researchers,  participants  learned  about  ethics  standards, 
including  the  right  to  withdraw  from  the  study  without  having  to  withdraw  from  the  program.  Each  participant 
chose  a screen  name  or  alias,  and  data  were  recorded  for  these  aliases.  Where  privacy  dictated,  the  academic 
researcher  occasionally  chose  to  use  an  additional  naming  convention  as  well.  Those  participants  who  did  leave 
the  study  have  given  their  permission  to  continue  using  their  data. 

Discourse  Analysis 

Discourse  analysis,  a qualitative  research  method  used  to  examine  the  peer  mentoring  sessions, 
consists  of  a number  of  approaches  to  analysis  of  verbal  interactions.  Discourse  analysis  was  agreed  upon  as  a 
method  for  the  study  by  the  participants,  who  saw  this  method  as  a means  of  investigating  empowerment.  This 
view  is  consistent  with  the  literature:  in  Language  and  Power , Fairclough  (1989)  invited  both  researchers  and 
lay  people  to  investigate  the  uses  of  discourse  strategies  to  take  and  consolidate  power. 

It  is  the  branch  of  discourse  analysis  known  as  conversation  analysis,  or  CA,  that  was  primarily  used  in 
this  study.  In  conversation  analysis,  audio  or  video  recordings  are  transcribed,  and  the  transcriptions  then 
analyzed,  with  structural  details  taking  precedence  over  content.  That  is,  rather  than  interpreting  data  and 
assigning  motives,  conversation  analysts  seek  information  about  the  structure  of  the  conversation.  Intonations, 
pauses,  interruptions  and  grammatical  structures  are  among  the  features  typically  coded  for. 

Conversation  analysis  was  pioneered  by  sociologist  Harvey  Sacks  in  the  1960’s;  his  work,  edited  by 
Gail  Jefferson  and  Emmanuel  Schegloff,  was  not  published  until  1992,  as  the  Lectures  on  Conversation  (Sacks, 
1992).  However,  long  before  this  publication  date.  Sacks,  Jefferson,  and  Schegloff  had  began  a collaboration 
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that  resulted  in  a series  of  works  that  defined  conversation  analysis.  These  works  included  the  Semiotica  article 
"Opening  up  Closings”  in  which  Schegloff  and  Sacks  proposed  the  concept  of  the  adjacency  pair  to  explain 
conversational  structures  such  as  questions  and  answers  (Schegloff  & Sacks,  1973)  as  well  as  the  Language 
article,  "A  Simplest  Systematics  for  the  Organization  of  Turn-Taking  for  Conversation"  (Sacks,  Schegloff  and 
Jefferson,  1974).  Working  with  Sacks'  material,  Gail  Jefferson  originated  a set  of  notations  for  marking  up  text 
that  was  published  with  the  1974  article,  and  that  has  remained  standard  among  conversation  analysts. 

In  addition  to  Sacks'  pioneering  work.  Sacks,  Schlegloff  and  Jefferson  built  on  the  work  of  Harold 
Garfinkel,  founder  of  the  ethnomethodology  school  of  sociology.  Ethnomethodology  focuses  on  the  study  of 
everyday  actions,  including  language  use,  and  takes  for  granted  that  ordinary  people  understand  what  they  are 
doing;  thus  it  is  by  nature  well  suited  to  a participatory  action  research  project  in  which  participants  are  co- 
researchers. Garfinkel  presented  the  principles  of  his  approach  in  Studies  in  Ethnomethodology,  citing 
transcripts  of  conversations  to  show  how  people  organize  their  shared  realities  using  "common  understandings" 
(Garfinkel,  1967,  p.  38).  In  other  words,  ordinary  people  can  and  do  understand  their  own  actions.  Like  the 
conversation  analysts,  Garfinkel  relied  on  observable  data,  using  only  as  much  context  as  was  necessary  to 
understand  the  data  and  expressing  a belief  that  only  those  involved  could  understand  the  whole  situation 
(Garfinkel,  1967). 

Also  related  is  the  work  of  Erving  Goffmann.  In  The  Presentation  of  Self  in  Everyday  Life,  Goffman 
showed  how,  whether  consciously  or  not,  conversationalists  act  out  roles  and  create  a ”front"-"that  part  of  the 
individual's  performance  which  regularly  functions  in  a general  and  fixed  fashion  to  define  the  situation  for 
those  who  observe  the  performance"  (Goffman,  1959,  p.22).  Conversation  analysis  draws  from  the  work  of 
Goffmann  in  that  both  reveal  the  ways  in  which  relationships  are  constructed  and  roles  created. 

Transcription  in  itself  has  come  to  be  viewed  as  a theoretical  process,  a viewpoint  introduced  by  Elinor 
Ochs  in  "Transcription  as  Theory”  (1979)  and  by  Carole  Edelsky,  in  "Who's  Got  the  Floor?"  (1981).  Central 
here  are  the  realizations  that  the  raw  data  are  not  the  same  as  the  transcribed  data;  that,  rather,  the  transcription 
is  a representation  of  the  data;  and  that  the  infinite  multitude  of  decisions  made  during  transcription  are  actually 
theoretical  decisions  reflecting  the  relative  importance  of  particular  discourse  features.  Two  researchers 
working  independently  would  not  make  the  same  theoretical  decisions;  and  a single  researcher,  writing  about 
the  same  text  at  different  times,  may  make  different  coding  decisions. 

It  cannot,  therefore,  be  assumed  that  reliance  on  observable  data  over  inference  in  conversation 
analysis  implies  that  the  data  are  wholly  objective.  The  ongoing  development  of  software  applications  that  can 
be  brought  to  bear  on  the  transcription  process  (for  example,  software  that  measures  pauses)  does  not  change 
this,  as  researchers  will  continue  to  make  the  important  decisions:  for  example,  pauses  in  a technology 
mentoring  session  may  have  to  do  not  with  hesitation  but  with  mentee  concentration  on  a technology  task. 
Where  there  is  reason,  a second  researcher  may  separately  transcribe  audio  or  video  recordings,  and  the  two 
versions  can  be  compared;  or,  a group  of  researchers  working  together  may  listen/view  and  transcribe  as  a 
group,  as  is  recommended  procedure  in  ethnographic  inquiry  and  interaction  analysis  (Jordan  & Henderson, 
1995).  However,  neither  process  is  necessarily  indicated  if  it  is  accepted  that  transcription  decisions  are 
properly  the  domain  of  the  researcher  (c.f.  Ochs,  1979;  Edelsky,  1981;  Lemke,  1998). 

A second  branch  of  discourse  analysis  useful  in  a participatory  action  research  study  is  critical 
discourse  analysis,  or  CDA,  in  which  discourse  is  examined  for  underlying  relationships  of  power.  In  the 
European  countries,  Foucault  is  thought  of  as  the  father  of  discourse  analysis  (e.g.  Fricke,  1999;  Diaz-Bone, 
2003),  in  part  for  work  published  in  his  Archeology  of  Knowledge  and  in  his  Discourse  on  Language , published 
together  in  English  in  1972.  Elsewhere,  Foucault  might  be  thought  of  at  least  as  the  godfather  of  critical 
discourse  analysis:  Foucault  began  the  Archeology  of  Knowledge  by  declaring  that  in  order  to  understand  any 
discourse  it  would  be  necessary  first  to  get  rid  of  all  preconceived  ideas  in  order  to  begin  anew,  and  then  to 
examine  relations,  including  relations  of  power. 

The  Floaters.org  study  is  closer  in  its  approach  to  another  critical  discourse  analyst,  Norman 
Fairclough.  Fairclough's  use  of  close  readings  of  text  resembles  that  of  conversation  analysis,  although  his 
intention  is  to  identify  and  uncover  power  relations.  In  Language  and  Power  (1989),  Fairclough  not  only 
identified  discourse  structures  that  give  and  take  power,  but  invited  others  (including  those  with  no  prior 
academic  background)  to  do  so  as  well. 

Participatory  Action  Research 

The  term  “action  research”  has  several  meanings.  In  educational  research,  action  research  might  be 
classified  into  two  major  branches,  practitioner  research  and  participatory  action  research  or  PAR.  In  both, 
research  is  carried  on  at  least  in  part  by  people  who  have  not  been  formally  trained  as  researchers,  but  who  have 
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the  advantage  of  being  directly  within  the  research  situation  and  so  have  access  to  an  unusually  high  level  of 
information.  Practitioner  research  in  education  rests  on  the  belief  that  teachers  (“practitioners”)  can  participate 
in  research,  generally  via  self  study,  rather  than  being  consumers  of  research  only.  In  “participatory  action 
research”  on  the  other  hand,  the  object  is  goal-oriented  social  justice,  as  guided  by  those  affected.  In  PAR,  a 
researcher  or  researchers  may  enter  the  situation  with  the  intent  of  forming  a participatory  research  group.  The 
Floaters.org  study  utilizes  the  PAR  approach:  the  initial  impulse  came  from  a researcher,  who  invited  a 
counterpart  in  an  affected  community  to  join  together  in  investigating  technology  integration  by  distributing 
donated  computers. 

Further  confusion  as  to  the  nature  of  action  research  arises  because  participatory  investigation  or 
participatory  action  research  has  been  used  widely  by  such  entities  as  the  World  Bank  (Narayan,  Patel,  Schafft, 
Rademacher  & Koch-Schulte,  2000).  Such  entities  generally  have  an  a priori  interest  in  showing  immediate 
financial  gains  for  participants,  whereas  true  participatory  action  research  centers  on  the  identification  of 
participant  values  prior  to  the  taking  of  action  (Fals  Borda,  1995;  Sohng,  1996). 

Participatory  action  research  has  been  used  in  the  United  States  in  the  establishment  of  support  groups 
for  sufferers  from  AIDS  (Glasser  & Bridgman,  1999)  and  in  the  investigation  of  the  efficacy  of  prison 
education.  (Fine,  Boudin,  Bowen,  Clark,  Hylton,  Migdalia,  Missy,  Rivera,  Robers,  Smart,  Torre  & Upegui, 
2001).  In  education,  the  work  of  Paolo  Freire  can  also  be  seen  as  participatory  action  research,  in  that  Freire 
mobilized  participants  to  take  control  of  their  education  (Freire,  1970,  1994,  1998). 

In  participatory  action  research,  participants  are  co-researchers,  with  an  equal  voice  in  all  decisions. 
The  method  itself  therefore  is  also  open  to  further  refinement.  Before  the  Floaters.org  study  began,  five  years’ 
of  group  work  on  the  project  had  resulted  in  three  principles  against  which  all  decisions  were  tested:  equality  of 
voice;  consensus -based  decision  making;  and  revolving  authority,  in  which  those  who  are  expert  in  a particular 
area  may  step  forward  and  take  charge,  so  long  as  those  who  are  interested  in  the  area  may  also  participate. 

In  interactions  outside  of  the  group,  “Nothing  about  us  without  us”  is  the  credo  of  grassroots 
participatory  action  research,  as  documented  on  the  Homeless  People’s  Network,  the  sister  website  and  mailing 
list  for  the  Floaters.org  project  (c.f.  also  Charlton,  2000).  “Speak  truth  to  power,”  is  a related  imperative, 
implying  that  participants  must  at  every  step  of  the  way  open  to  stating  their  opinion  rather  than,  for  example, 
avoiding  the  situation  (Carey  et  al,  1955;  Kennedy  Cuomo  & Adams,  2004). 

With  these  guiding  principles  in  participatory  action  research,  project  members  take  an  ethical  stance 
that  is  every  bit  as  difficult  to  implement  as  the  most  rigorous  of  quantitative  methods;  in  both,  at  the  end  of  the 
study,  the  researchers  know  that  their  findings  are  trustworthy  to  the  extent  that  the  method  has  been  properly 
followed. 


Self  Efficacy 

"Perceived  self  efficacy"  refers  to  people's  optimistic  beliefs  about  their  ability  to  reach  their  goals 
(Bandura,  1977,  2001).  Hackett  and  Betz  have  stated  that  one  of  the  most  useful  concepts  in  modern  psychology 
is  Albert  Bandura's  concept  of  self-efficacy  expectations,  for  self  efficacy  has  been  shown  to  affect  human 
development  (Hackett  & Betz,  1981;  Betz  & Hackett,  1981;  Bandura,  1977;  Bandura,  2001).  Changes  in  self 
efficacy  have  four  sources,  according  to  Bandura:  mastery  experiences,  vicarious  experiences  modeled  by 
others,  verbal  persuasion,  and  people's  own  physiological  indicators  or  somatic  and  emotional  states  resulting 
from  the  attempt  to  achieve.  Investigating  these  four  sources,  Bandura  found,  for  example,  that  verbal 
dissuasion  is  easier  to  accomplish  than  verbal  persuasion;  that  both  positive  and  negative  moods  may  affect  self 
efficacy,  as  can  pain;  and  that,  in  a learning  environment,  teacher  self  efficacy  can  negatively  or  positively 
impact  student  self  efficacy  (Bandura,  1994,  1995,  1997,  2001).  Bandura’s  work  with  self  efficacy  was  a part  of 
his  pioneering  work  that  made  a place  for  cognition,  rather  than  adhering  to  the  older  “black  box”  theories  of 
psychology.  In  thus  allowing  for  the  thought  processes  of  the  individual,  Bandura  created  a concept  which  is 
more  appropriate  to  grassroots  PAR  investigation,  which  also  places  emphasis  on  the  individual  participant, 
than  earlier  theories. 

Self  efficacy  is  domain  specific.  Bandura  states  in  his  Guide  to  the  Construction  of  Self-Efficacy  Scales 
(2001)  that  is  important  to  use  a domain -specific  measure  rather  than  a general  measure  of  efficacy,  as  general 
measures  are  too  ambiguous  to  be  meaningful.  Self  efficacy  can  transfer  across  domains,  and  powerful 
experiences,  in  particular,  can  effect  changes  across  many  domains  (Bandura,  1994,  1995,  1997,  2001). 

With  regard  to  the  study  of  self  efficacy  with  low  income  populations,  early  research  conducted  by 
CTCNet  (CTCNet,  acc.  2003)  identified  a number  of  benefits  to  users,  one  of  which  was  personal  efficacy 
(Mark,  Cornebise  & Wahl,  1997).  In  subsequent  CTCNet  studies,  this  benefit  was  dropped  as  a concept  of 
interest  (Chow  et.  al.,  2000),  but  a more  domain -specific  self-efficacy  concept  may  be  worth  a closer 
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examination. 

While  career  goals  are  not  the  only  objective  of  technology  integration,  they  are  a concept  of  interest 
among  the  Floaters.org  population,  as  determined  by  the  participatory  research  group.  "Work  self  efficacy"  or 
"Occupational  self  efficacy"  refers  to  the  belief  that  one  can  succeed  in  a particular  job.  The  concept  of 
occupational  self  efficacy  itself  was  introduced  in  pioneering  work  published  in  1981  by  Gail  Hackett  and 
Nancy  Betz  (Hackett  & Betz,  1981;  Betz  & Hackett,  1981).  In  these  works,  Betz  and  Hackett  applied  the 
concept  of  self  efficacy  to  career  counseling  and  discovered  gender  differences  among  with  regard  to 
traditionally  gendered  occupations,  in  particular,  men  had  much  stronger  self  efficacy  scores  for  traditionally 
male  careers  (Betz  & Hackett,  1981;  Hackett  & Betz,  1981.) 

Following  Bandura's  model,  Hackett  and  Betz  created  the  Occupational  Self-Efficacy  Scale  (OSES)  in 
the  form  of  a Likert  scale  from  one  to  ten  for  20  occupations;  respondents  were  asked  to  select  their  answers 
based  first  on  how  certain  they  were  of  being  able  to  complete  the  education  or  training  for  each  occupation, 
and  then  on  how  certain  they  were  of  being  able  to  carry  out  the  job  duties  (Betz  & Hackett,  1998).  A revised 
version  of  their  1981  Manual  for  the  Occupational  Self-Efficacy  Scale  was  made  available  online  in  1998  (Betz 
& Hackett,  1998).  Typically  the  scale  is  revised  by  the  researcher  to  offer  relevant  occupations  for  the 
population  of  interest  (Hackett,  personal  communications,  2001,  2003);  in  this  PAR  study,  the  participants 
themselves  chose  the  relevant  occupations.  The  scale  can  be  administered  in  terms  of  the  training  aspect,  in 
terms  of  the  job  duties  aspect,  or  with  both  aspects  included. 

Method 


Participants 

Participants  in  the  group  (n=184  at  the  time  of  the  study)  comprise  a “snowball’'  sample  (Babbie, 

1998)  in  that  new  participants  were  recruited  by  project  members,  each  of  whom  was  asked  to  choose  someone 
else  to  teach.  For  the  study,  37  primary  participants  were  lent  recycled  computers,  standardized  to  the  extent 
possible.  Data  were  collected  over  the  course  of  a year  for  each  of  the  37.  Each  primary  participant  represented 
a different  family  except  where  there  were  two  or  more  computers  in  a family;  in  these  cases,  each  primary  user 
of  a computer  was  also  a primary  participant  in  the  study.  All  of  the  37  volunteered  for  the  project,  and  all  were 
over  the  age  of  eighteen. 

Each  section  of  the  study  drew  on  a different  subset  of  participants  as  appropriate: 

Fourteen  of  the  37  took  part  in  the  taped  peer  mentoring  sessions.  The  taped  sessions  used  for  the  study 
were  theoretically  sampled,  that  is,  they  were  chosen  in  order  to  provide  as  wide  a range  as  possible  in  terms  of 
age,  level  of  experience  in  the  program,  gender,  and  level  of  education. 

All  participants  were  invited  to  optional  weekly  research  meetings,  where  data  were  collected  in  the 
form  of  field  notes  and  observations  regarding  changes  in  the  structure  of  the  Floaters.org  project  during  the 
study. 

Twenty-two  members  agreed  to  take  part  in  the  self  efficacy  portion  of  the  study;  of  these,  seventeen 
took  part  in  both  administrations  of  the  self-efficacy  scale,  at  approximately  four-month  intervals,  while  five 
took  part  only  in  the  first  administration.  The  22  were  self -selected  from  the  37  primary  participants. 

While  research  decisions  (e.g.  aspects  studied  and  methods)  were  made  by  the  participants  after  group 
study  of  research  methods,  and  while  participants  took  part  in  other  aspects  of  the  research  as  appropriate  (e.g., 
participants  selected  the  careers  to  appear  on  the  Occupational  Self  Efficacy  Scale),  a trained  researcher 
conducted  the  analysis,  with  constant  member  checks  wherein  participants  were  asked  to  comment  on  portions 
of  the  analysis. 

Data  Sources 

Data  sources  included  the  following: 

• Records  of  the  computers  and  the  work  done  with  them 

• Field  notes,  observations,  and  qualitative  memos 

• Eight  videotaped  mentoring  sessions,  theoretically  sampled  so  as  to  provide  the  widest  possible  variety  of 
mentors  and  mentees 

• Field  notes  and  observations  from  focus  group  sessions 

• Scores  from  the  two  administrations  of  the  Occupational  Self  Efficacy  Scale 

• Responses  to  followup  questions  asked  of  those  who  took  part  in  the  mentoring  study 
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Procedure 

Recycled  Macintosh  computers  with  a standard  set  of  applications  were  distributed  to  thirty-seven 
primary  participants,  along  with  self-study  materials  and  resources  for  feedback. 

Eight  peer  mentoring  sessions,  theoretically  sampled  to  provide  the  widest  possible  range  in  education, 
experience,  age,  and  gender,  were  videotaped  by  the  researcher. 

The  Occupational  Self-Efficacy  Scale  was  administered  at  approximately  four-month  intervals  to  a 
subset  of  twenty-two  self-selected  mentors  and  mentees  in  order  to  determine  any  changes  in  attitude  towards 
computer-related  jobs.  Seventeen  of  the  twenty-two  were  available  to  take  the  self-efficacy  instrument  a second 
time;  five  took  it  only  once. 

Weekly  focus  group  sessions/research  group  sessions  were  held,  and  follow-up  interviews  were 
scheduled  during  the  last  three  months. 
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Figure  1.  Timeline  for  the  study.  The  equivalent  of  the  recommended  year  for  ethnographic  data  collection  was 
allotted  to  data  collection  for  the  study,  along  with  a second,  also  recommended,  year  for  analysis  and  writing. 


Analysis 


Peer  Mentoring  Sessions : A Discourse  Analysis 

The  first  of  the  three  aspects  studied  was  that  of  the  mentor/mentee  relationship.  For  the  discourse 
analysis  portion  of  the  study,  eight  theoretically  sampled  peer  mentoring  sessions  were  taped,  transcribed,  and 
coded  for  discourse  features  in  order  to  examine  the  participant  roles  of  mentor  and  mentee.  The  sessions  were 
selected  via  the  theoretical  sampling  process  in  order  to  provide  as  wide  a range  as  possible  regarding  age,  level 
of  experience  in  the  program,  gender,  and  level  of  education.  The  mentoring  pairs  were  videotaped  during 
sessions,  and  the  sessions  were  transcribed  and  marked  up,  then  coded,  using  modified  conventions  based  on 
the  work  of  Gail  Jefferson  (Sacks,  Schegloff  & Jefferson,  1974;  Jefferson,  1984).  It  is  standard  practice  for  the 
researcher  to  adjust  the  transcription  conventions  to  the  requirements  of  the  researcher’s  interests  and  of  the  text 
at  hand,  so  the  conventions  given  here  vary  from  those  originated  in  1974.  Data  were  coded  line  by  line,  in 
constant  comparison  analysis  (Glesne  & Peshkin)  until  categories  were  found  and  then  saturated,  that  is,  until 
all  new  data  fit  into  already  discovered  categories.  Transcription  conventions  used  for  marking  up  the  text  can 
be  found  in  Appendix  A. 

Scalability/Changes  to  the  Program 

The  second  aspect  studied  was  that  of  scalability  of  the  project.  At  the  beginning  of  the  study,  the 
project  was  scaled  up  in  size.  Rather  than  accepting  new  members  one  at  a time  as  mentors  became  available, 
twenty  new  families  were  accepted  at  once.  As  the  program  was  modified  in  order  to  accomplish  this  expansion 
of  the  program  and  to  provide  ways  to  collect  data,  the  opportunity  arose  to  track  answering  changes  produced 
by  the  participants  responsible  for  the  program  structure.  Most  important  of  these  changes  was  the 
establishment  of  a weekly  focus  group  that  any  participant  (n=184)  could  attend  and  that  served  as  a means  of 
communicating  concerns  as  well  as  to  continue  the  work  of  shaping  the  program.  Field  notes  of  these  focus 
group  meetings  were  collected  by  the  researcher  and  analyzed,  again  via  coding  and  constant  comparative 
analysis,  in  order  to  investigate  how  the  role  of  shaper  and  developer  of  the  program  played  out  during  this 
time. 

Occupational  Self  Efficacy 

To  examine  occupational  self-efficacy,  the  Occupational  Self-Efficacy  Scale  or  OSES  was  revised  in 
accordance  with  guidelines  set  by  Bandura  (2001)  and  advice  given  by  Hackett  (personal  communications, 
2001,  2003).  The  OSES  is  typically  revised  by  the  researcher,  who  selects  occupations  that  are  relevant  to  the 
group  under  study;  here,  the  participants  were  asked  to  choose  the  occupations.  Rather  than  limit  the  list  of 
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occupations,  they  chose  to  use  them  all,  adding  eight  more  technology  occupations  and  the  occupation  of 
President. 

For  this  aspect  of  the  study,  four  smaller  questions  were  formulated: 

1.  Will  there  be  a significant  difference  between  means  for  technology-related  occupations  and  non-technology 
related  occupations  within  the  first  administration  of  the  instrument? 

2.  Will  there  be  a significant  difference  between  means  of  technology-related  occupations  and  non -technology 
related  occupations  within  the  second  administration  of  the  instrument? 

3.  Will  there  be  a significant  difference  between  means  of  non-technology  occupations,  between  the  two 
administrations  of  the  instrument? 

4.  Will  there  be  a significant  difference  between  means  of  technology-related  occupations,  between  the  two 
administrations  of  the  instrument? 

Scores  on  the  self-efficacy  scales  were  analyzed  for  each  of  the  four  smaller  research  questions  by 
running  a paired  samples  t-test  to  test  for  difference  of  means.  Follow-up  questions  were  generated  based  on  a 
pattern  analysis  of  the  results,  and  participants  were  contacted  and  interviewed. 

Results 

Peer  Mentoring  Sessions:  A Discourse  Analysis 

Analysis  of  the  peer  mentoring  sessions  showed  that  peer  mentors  developed  and  used  sophisticated 
teaching  strategies  and  that  these  strategies  were  similar  among  experienced  mentors,  regardless  of  education 
levels.  Coding  yielded  two  sets  of  mentor  discourse  strategies  that  had  to  do  with  verbal  contributions  of  the 
rnentee  to  the  session.  Differences  were  found  in  the  use  of  these  strategies  between  experienced  and  beginning 
mentor  discourse:  expert  mentors  alternated  the  two  types  of  strategies,  thereby  encouraging  mentee 
contributions  while  advancing  the  instruction.  On  the  other  hand,  features  found  in  the  discourse  of  experienced 
mentors  were  similar  regardless  of  educational  level:  members  with  and  without  a high  school  diploma  were 
equally  expert  in  mentoring  if  they  had  an  equal  amount  of  mentoring  experience. 

One  set  of  strategies  encouraged  mentee  participation:  these  strategies  included  such  features  as  questions, 
problem  statements,  and  off-topic  remarks.  This  problem  statement  by  the  mentor,  for  example,  is  followed  by  a 
mentee  response: 

->Mentor:  30.  I’ve  never  seen  the  mouse 

3 1 . connected  to  the  cord  before . 

Mentee  32.  (.)  Then  nobody  can  steal  them. 

The  second  set  of  strategies  included  such  discourse  features  as  interrupting,  ignoring  mentee  contributions,  and 
speeding  up  the  rate  of  mentor  speech.  Here,  the  mentor  interrupts,  thereby  retaining  control  of  the  instruction: 

->  Mentee:  23.  Let  me  show  [you  ( ) 

->  Mentor:  24.  [And  then  you  can,  now  if  you  want  to  do. 

25.  ° Let  me  show  you. ° 

Scalability/Changes  to  the  Program 

Participants  modified  the  Floaters.org  program  further  during  the  study,  primarily  increasing 
communication  opportunities.  The  first  modification,  in  which  participants  requested  that  the  focus  and  research 
group  meet  weekly,  and  that  it  be  immediately  opened  to  mentors  and  mentees  alike,  provided  a regular 
opportunity  for  participants  to  bring  up  their  concerns  and  to  suggest  changes.  Participants  successfully 
maintained  the  basic  premises  of  the  program  (equality  of  voice,  consensus -based  decision  making,  and 
revolving  authority)  throughout  the  mixed  method  study:  no  mean  feat,  for  as  the  group  learned,  in  a research 
situation  methods  can  easily  deviate  from  grassroots  principles  unless  there  is  constant  attention  to  the 
principles.  A second  major  modification  was  that  of  the  addition  of  a fourth  principle  to  group  interactions: 
speaking  truth  to  power  within  the  group  (as  well  as  outside  of  it)  rather  than  walking  away.  This  addition  to  the 
internal  principles  of  participatory  action  research  was  needed  since  many  participants  (as  documented  in  the 
self  efficacy  interviews)  perceived  of  their  power  as  limited  in  some  senses,  but  strong  in  that  they  stated  that 
they  could  always  walk  away  from  a situation  in  which  they  were  not  being  treated  fairly.  Encouragement  to 
speak  out  in  the  group  over  perceived  slights  was  the  solution  to  losing  participants  over  such  problems. 
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Occupational  Self  Efficacy 

Results  of  the  self-efficacy  questionnaire  were  inconclusive,  though  a major  history  effect  may  have 
had  some  impact  on  these  results.  The  study  was  interrupted  by  the  events  of  September  11,  2001,  and  in 
follow-up  interviews  as  well  as  in  focus  group  observations,  some  participants  indicated  that  these  events  had 
changed  their  opinions. 

One  significant  difference  was  found,  for  the  exploratory  question.  Will  there  be  a significant 
difference  between  means  for  technology-related  occupations  and  non -technology  related  occupations  within 
the  first  administration  of  the  instrument? 

For  the  difference  between  means  of  technical  and  non-technical  occupations  in  the  first  administration 
of  the  OSES,  the  p value  was  .047.  As  the  sample  size  is  small,  a non-parametric  test,  the  Wilcoxon  Signed 
Ranks  test  was  also  run,  and  here  the  p value  was  .049,  leading  to  the  same  conclusion. 

This  difference  disappeared  in  the  second  administration  of  the  OSES,  where  scores  for  non-technology 
occupations  rose,  although  not  significantly.  Standard  deviations  for  technology-related  occupations, 
particularly  in  the  second  administration,  were  higher  than  for  non-technology-related  occupations: 


Table  1 Standard  Deviations 


Non-technology  1 Technology  1 Non-technology  2 Technology  2 


1.6949 


1.8254 


1.6474 


2.0261 


These  results  are  somewhat  counter-intuitive  as  it  might  have  been  expected  that  technology  self 
efficacy  would  increase  after  participants  received  their  computers.  In  order  to  investigate  this  further,  responses 
to  the  scale  were  first  graphed  and  patterns  of  responses  were  analyzed,  in  order  to  generate  follow-up 
questions.  Open-ended  follow-up  qualitative  interviews  were  then  held. 

Three  types  of  typical  patterns  were  found:  one  subset  reached  the  ceiling  in  technology  self-efficacy 
either  in  the  second  administration,  or  in  both  administrations:  these  scores  either  rose,  or  stayed  the  same. 

Statements  from  this  group  in  follow-up  interviews  indicated  that  these  participants  wished  to  make  it 
clear  that  they  could  do  anything  if  they  had  the  opportunities  for  education  and  success  that  others  have.  These 
members  felt  that  they  were  capable  of  completing  the  education  for  occupations  requiring  high  levels  of 
education,  but  felt  others  in  society  were  unaware  of  their  ability. 

In  another  subset,  scores  fell,  sometimes  radically.  Interviews  with  this  group  indicated  that  after 
September  1 1,  some  participants  lost  interest  in  their  work  with  the  computer,  and  some  lost  faith  in  their  ability 
to  prepare  for  computer-related  jobs. 

With  a third  group,  response  levels  varied  based  on  the  specific  type  of  computer  career,  either  in  both 
administrations  or  in  the  second.  That  is,  a participant  might  score  high  on  a career  involving  computer  art,  but 
low  on  a career  involving  programming.  In  follow-up  interviews,  these  participants  indicated  that  their  greater 
understanding  of  technology  careers  had  led  them  to  give  more  precise  answers,  sometimes  resulting  in  an 
overall  lower  score. 


Looking  at  the  Participatory  Process 

With  regard  to  participant  goals,  localized  best  practices  were  found  regarding  technology  integration. 
Financial  goals  remained  problematic,  but  other  goals  defined  by  participants  as  pertaining  to  a better  life  were 
met  to  a degree.  As  one  participant  stated,  “I  will  stay  as  long  as  it  feeds  my  spirit.” 

Also  documented  in  the  focus  group  sessions  and  noted  in  the  Changes  to  the  Program  section  is  the 
refinement  of  the  participatory  action  research  process  over  the  course  of  the  study. 

Participatory  studies  may  more  commonly  limit  methods  to  ethnography:  in  this  study,  other  methods 
were  used  as  well,  but  introduced  with  care  and  always  with  participant  understanding  and  consent.  For 
example,  the  research  group  voiced  concerns  that  quantitative  methods  would  be  disemp  owering,  and  a 
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discourse  analysis  has  been  proposed  to  determine  if  the  self-efficacy  scale  is  less  empowering  with  this 
population  than  an  open-ended  questionnaire. 

As  a method,  discourse  analysis  was  not  seen  as  disempowering,  as  the  group  was  introduced  to  it  via 
Fairclough’s  call  for  lay  attention  to  discourse  as  a means  of  taking  power  (Fairclough,  1989.)  The  origins  of 
conversation  analysis  being  to  some  extent  in  ethnomethodology,  a field  in  which  ordinary  people  are  seen  as 
able  to  understand  their  own  discourse,  further  makes  this  method  approachable  to  discourse  analysis.  Finally, 
the  place  of  self  efficacy  in  psychology  as  introducing  the  importance  of  cognition  makes  this  concept  also 
appropriate. 


Discussion  and  Future  Research 

Generalizability  in  the  traditional  meaning  of  the  word  is  not  the  purpose  of  a study  that  takes  a 
qualitative  perspective;  instead,  theory  is  generated  in  the  form  of  hypotheses  based  directly  on  data.  In  such  a 
study,  sufficient  detail  is  ideally  presented  for  readers  to  be  able  to  make  the  decision  as  to  whether  or  not  the 
findings  may  also  generalize  to  particular  situations  with  which  the  researcher  is  familiar.  Such  detail  adds  to 
the  credibility  of  the  study.  In  this  brief  paper,  detail  is  necessarily  necessity  limited:  further  papers  are  planned, 
and  in  the  meantime  it  is  hoped  that  the  detailed  presentation  here  of  the  participatory  process  as  used  in  this 
study  will  encourage  the  reader  to  entertain  the  assumption  that  the  rest  of  the  study  may  likewise  be  credible. 

It  is  also  important  for  credibility  that  rapport  and  “buy-in”  on  the  part  of  participants  can  be  shown. 
This  study  sought  such  rapport  and  buy-in  by  following  participatory  action  research  principles.  As  all 
participants  are  co-researchers  with  an  equal  voice  in  decision  making,  rapport  was  a natural  outcome. 

Credibility  is  further  enhanced  by  triangulation,  or  the  use  of  multiple  methods  and/or  data  sources, 
especially  when  the  findings  of  the  methods  and  sources  converge,  as  they  do  here.  In  terms  of  the  purposes  of 
the  project  and  of  the  study,  a project  using  specific  means  of  technology  integration  (participatory  research, 
peer  mentoring,  focus  groups,  home  -based  technology  integration)  was  examined  in  differing  ways  (discourse 
analysis,  field  notes,  and  a self-efficacy  scale)  to  yield  converging  results. 

The  results  point  to  the  usefulness  and  potential  for  success  of  peer  mentoring,  a process  that  is 
eminently  affordable  as  a means  of  instruction.  The  discourse  strategies  can  be  studied  further;  one  next  step  for 
this  project  will  be  to  incorporate  the  strategies  into  mentor  training. 

Within  the  participatory  process,  it  is  interesting  to  note  that  grassroots  participants,  after  five  years  of 
work  with  the  process,  were  able  to  transform  the  project  to  meet  the  challenges  of  scalability  and  indeed  to 
further  refine  the  participatory  process.  Further  study  of  the  nature  of  power  and  empowerment  for  such  a group 
appears  to  be  in  order. 

As  mentioned  earlier,  participants  themselves  have  suggested  a close  look  at  the  discourse  of  those 
who  are  taking  the  OSES,  in  order  to  determine  if  the  OSES  is  empowering  or  disempowering  as  an  educational 
experience 

Another  direction  for  study,  and  currently  underway,  is  the  further  development  of  the  OSES  in  the 
direction  of  a Technology  Occupational  Self-Efficacy  Scale,  Important  here  will  be  the  identification  of  sets  of 
related  technology  occupations  that  are  of  varying  difficulty,  so  as  to  satisfy  Bandura's  requirement  that  items 
on  the  scale  present  varying  levels  of  challenges  (Bandura,  2001).  To  take  an  example  from  the  current  version 
of  the  scale,  "video  game  tester"  and  "video  game  designer"  require  varying  levels  of  training  or  education. 

Since  technology-related  occupations  exist  across  various  subdomains-programming,  networking,  and  art,  for 
example-it  will  be  important  to  identify  the  various  technology-related  domains.  A related  avenue  of  future 
research  will  be  the  further  development  of  an  instructional  intervention  that  describes  the  technology  domains. 

Appendix  A 

Transcription  Conventions  Used  in  the  Floaters.org  Study 
Markup  is  based  on  the  work  of  Gail  Jefferson  (1974,  1984). 

Markup  Definition/Samples 

[ Left  bracket:  beginning  of  overlap  or  interruption 

S.  Yah  I was  in  one  of  those  technology  classes  [or 

A.  [Computer  class? 
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Right  bracket  (optional):  end  of  overlap  or  interruption 


= Equal  signs:  no  break  or  pause. 

= Two  equal  signs  indicate  no  break  between  the  lines.  One  equal  sign  may  be  lined  up  above 

the  other;  or,  one  may  appear  at  the  end  of  one  line  and  one  at  the  beginning  of  the  next. 

A.  one  day= 

S.  =yah  (.) 

T. 

Dash:  speech  is  cut  off. 

A.  All  in  the  box.  It  all  stays  in  the  box- 

no  ! Get  out!  Can  anybody  hear  me? 

yah  Underscoring  denotes  a louder  voice  or  other  stress,  as  in  a slight  rising  inflection. 

U.  Yea::h,  Pe:te! 

dit  Bolded  text  denotes  an  even  louder  voice  or  other  form  of  exaggeration: 

P.  dit  dit  dit  dit 

F.  That  said  it  closed  down  (.)  i:m  properly= 

::  Colons  indicate  prolongation  of  the  prior  sound.  The  longer  the  colon  row,  the  longer  the  prolongation. 

N.  She  lo::ves  using  the  computer. 

< Angle  brackets  (carets,  greater  than  signs)  surrounding  text  and  pointing  inward:  speech  is 

said  more  quickly  relative  to  other  utterances. 

S.  >very  cool  very  cool< 

o Angle  brackets  (carets,  greater  than  signs)  surrounding  text  and  pointing  outward:  speech  is 

said  more  slowly  relative  to  other  utterances. 

<and  you  ca::n’t  kill  it> 

( ) Empty  parentheses  or  with  the  word  "inaudible"  indicate  something  not  heard  by  the 

transcriber. 

(yah?)  Parenthesized  words  are  guesses  on  the  part  of  the  transcriber. 

° ° Degree  signs  indicate  something  said  very  quietly. 
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F. 


’Let  me  show  you. 


{ } Curly  brackets  indicate  context. 

{Context:  mentor  is  looking  at  computer. } 


((  ))  Doubled  parentheses  contain  descriptions  of  sounds  other  than  speech. 

Pauses  are  measured  approximately. 

New  lines  and  micropauses  (.)  are  about  .1  second. 

A number  within  parentheses,  such  as  (2.5),  is  also  approximate. 

Some  pauses  were  the  result  of  engagement  with  the  computer.  Where  this  engagement  was  audible,  it  was 
transcribed  with  an  explanation  in  double  parentheses,  as  for  example  ((clicking)). 

Silences  longer  than  a few  seconds  were  coded  by  approximate  length  or  by  an  explanation  within  double 
parentheses:  ((extended  silence)). 
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Background 

The  use  of  animation  and  audio  as  a virtual  panacea  for  everything  from  advertising  to  educational 
videos  and  instruction  has  created  the  presumption  that  any  materials  that  use  them  "must  be  better!’  Now  doubt 
that  the  addition  of  animation  can  improve  message  delivery  on  a number  of  scales,  but  the  use  of  animation 
regardless  of  message  and  with  little  concern  for  systematic  placement  may  be  causing  more  harm  than  good. 
Combined  with  the  increasing  prevalence  of  computing  technology  and  increasing  ease  of  development  on 
standard  workstations  the  integration  of  animation  in  web-based  instruction  is  a more  realistic  possibility.  This 
study  explores  the  effect  of  animation  on  higher  order  educational  objective  achievement  in  a web-based,  self- 
paced  programmed  instructional  unit  on  the  human  heart  and  its  functions  for  undergraduate  students  with 
majors  outside  of  the  life  sciences. 


Introduction 

In  instruction  the  use  of  pictorial  media  has  long  been  considered  to  be  an  important  instructional 
variable  supported  by  a number  of  theoretical  considerations.  Kulik  and  Kulik  ( 1985)  reported  that  computer- 
based  instruction  enhances  learning  and  fosters  positive  attitudes  toward  instruction  with  college  students. 
Unfortunately  most  of  the  studies  that  have  been  carried  out  examining  animation  in  computer-based  and  web- 
based  instruction  have  been  cursed  with  confounding  and  poor  designs.  (Rieber,  1991;  Dwyer  1978;  Park  & 
Hopkins  1993)  There  are  various  reasons  for  the  conflicting  results  and  inconclusive  results.  Dwyer  (1978) 
pointed  out  that  one  of  the  major  problems  with  media  research  was  that  it  didn’t  describe  the  types  of  learning 
tasks  or  objectives  that  it  expected  the  participants  to  achieve.  Sadly  upon  review  of  available  literature  dealing 
with  animation  in  computer-based  and  web-based  instruction  this  still  appears  to  be  the  case.  In  the  subsequent 
literature  review  this  author  makes  a case  that  not  much  has  changed  in  the  past  25  plus  years  in  media  research 
since  Dwyer’s  observation.  Further  the  questions  of  effectiveness  in  practice  and  convergence  of  theory  remain 
largely  inconclusive. 

Definition 

With  a lack  of  definition  of  terms  being  one  of  the  major  criticisms  of  the  research  reviewed  in  the 
subsequent  pages  the  researchers  will  explicate  their  operating  definitions  for  animation.  Animation  is  a 
sequence  of  images  played  in  rapid  succession  such  that  to  the  human  eye  the  result  is  apparent  motion  (Park  & 
Gittelman,  1992)  and  is  generally  used  in  instructional  materials  for  one  of  three  purposes:  attention-gaining  / 
attention-direction,  presentation,  and  practice  (Rieber,  1990). 

Literature  Review 

Paivio’s  dual  coding  theory  (DCT)  asserts  that  images  and  verbal  processes  together  determine 
learning  and  memory  performance  (Clark  & Paivio,  1991).  According  to  Paivio’s  (1971,  1986)  and  later  Clark 
and  Paivio’s  (1991)  explanation,  using  the  information  processing  model  and  the  spread  of  activation  in  the 
brain,  the  links  between  verbal  and  non-verbal  symbolic  storage  can  trigger  each  other,  “...this  spreading 
activation  results  in  complex  patterns  of  arousal  among  the  representations  in  the  network.”  (Clark  & Paivio, 
1991,  p.  154)  Further,  Clark  and  Paivio  propose  that  both  types  of  mental  representations  have  dedicated 
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channels  for  the  processing  and  encoding  of  information.  Tulving  (1976)  suggests  that  information  can  be 
processed  on  several  levels  in  parallel  processes  and  a logical  extension  of  this  using  information  processing  as 
a guide  is  that  the  parallel  processing  can  aid  in  the  transfer  of  information  into  long-term  storage.  The  concept 
of  parallel  processing  is  not  new  to  cognitive  psychology,  and  will  be  generally  accepted  by  the  researchers. 
What  is  of  interest  to  the  researchers  stems  from  the  apparent  inconsistencies  in  the  literature  regarding  the  use 
of  animated  sequences  to  facilitate  learning  or  achievement. 

Animation  Studies  and  Findings 

Park  and  Hopkins  (1993)  suggest  that  educational  research  on  visual  displays  evolved  from  two 
distinct  camps:  the  behaviorist  with  work  by  Guthrie,  Skinner,  and  Thorndike  on  eliciting  desired  responses 
after  some  stimulus,  and  the  cognitive  with  work  by  Paivio  and  his  Dual  Coding  Theory  (DCT).  Tulving  and 
his  concept  of  3 major  types  of  memory:  episodic,  semantic,  and  procedural  would  also  fall  under  the  cognitive 
branch  in  this  division  as  well.  (Gredler,  2001,  p.  171) 

In  Rieber’s  1990  review  of  12  studies  spanning  25  years,  he  found  inconsistent  results  in  the  effects  of 
animation  on  achievement  and  learning  by  extension.  Similarly,  Park  and  Hopkins  (1993)  reviewed  25 
empirical  studies,  17  of  which  dealing  directly  with  computer  based  instruction  and  also  reported  mixed  results. 
Dwyer  and  Dwyer  in  a 2003  presentation  reviewed  5 animation  based  studies  using  similar  content  and  identical 
assessment  tools  on  a total  of  781  subjects,  and  found  only  three  cases  out  of  the  72  examined  where  animation 
showed  significant  benefits  over  static  visuals.  Owens  and  Dwyer  (2003)  in  an  unpublished  study  actually 
found  animation  to  be  less  effective  than  static  visuals  at  higher  levels  of  learning. 

In  a study  published  in  1988  by  Reiber  and  Hannafin  looking  at  the  effects  of  animated  or  textual 
orienting  activities  on  learning  in  computer  based  instruction  with  fourth,  fifth,  and  sixth  graders,  they  report 
that  neither  text -based  or  animated  activities  were  powerful  influences  on  learning.  This  studies  content  was 
based  on  Newton’s  laws  of  motion,  and  the  authors  report  that  a 24  item  posttest  was  administered  with  a KR- 
20  reliability  of  .83  overall.  While  not  explained  they  also  report  that  validity  was  established  through  expert 
review  by  independent  science  teachers. 

In  a study  published  one  year  later  in  1989,  Reiber  ran  another  factorial  study  3x2x2  looking  at  more 
factors  comparing  graphic  type  (animation,  static  graphic,  not  graphic)  and  text  type  (text,  no-text)  and  practice 
type  (relevant,  irrelevant).  Additionally  within  subject  he  examined  factual  verses  application  objectives  and 
near  verses  far  transfer.  The  overall  reliability  of  the  improved  dependent  measures  was  .91 . In  this  study  “The 
lack  of  main  effects  among  the  embedded  elaboration  conditions  was  surprising.”  (Reiber,  1989,  p.  439)  Two 
years  later  Reiber  (1991)  ran  a simple  version  of  the  study  looking  only  at  graphic  type  (static  graphics  verses 
animated  graphics)  and  practice  type  (simulation  verses  simulation  with  questions)  found  significant  effects  in 
favor  of  animation  for  near  transfer  on  incidental  and  intentional  questions. 

In  a 1998  animation  study  conducted  by  Park  using  a computer  based  instructional  unit  on  electronic 
circuit  repair  he  reports  that  static  graphics  with  motion  cues  can  be  used  instead  of  full  graphical  animation  as 
they  both  were  equally  effective  on  the  performance  and  transfer  tests  that  were  administered.  Will  this  finding 
may  well  be  true,  Park  did  not  identify  the  level  of  educational  objectives  measured  or  report  any  reliability  or 
efforts  to  support  validity  on  his  dependent  measures.  If  the  dependent  measures  were  measuring  factual  or 
conceptually  based  knowledge  than  the  finding  may  offer  some  direction  an  support  but  with  out  it  being 
reported  the  readers  are  left  to  their  own  devices  and  field  suffers  once  again. 

“Although  much  research  has  been  done  on  the  effectiveness  of  static  visuals  (Dwyer,  1978),  little 
research  has  conducted  on  animation’s  instructional  effects.  Empirical  data  that  are  available  are  inconsistent.” 
(Reiber,  1990,  p.  78)  In  the  almost  15  years  since  Reiber  wrote  this  statement,  little  has  changed.  More  studies 
have  been  run,  but  a general  consensus  has  yet  to  be  reached.  This  lack  of  consistency  stems  from  several 
different  sources  in  both  internal  and  external  validity  issues.  There  are  the  more  obvious  issues  of  poor  study 
design,  varying  and  insufficient  sample  sizes,  issues  with  content  relevance,  lack  of  systematic  process  for 
placement  of  treatments,  use  of  assessments  with  out  evidence  of  -or  even  reported  assessment  reliability  and 
content  validity.  Even  with  all  of  these  issues  surrounding  the  existing  literature,  one  of  the  most  egregious 
errors  revolves  around  the  lack  of  definition  of  the  types  or  methods  of  animation  used  and  the  levels  of 
instructional  objectives  that  were  being  addressed  in  the  studies.  The  failure  of  previous  researchers  to  provide 
reasoning  for  placement  of  the  interventions  casts  yet  more  doubt  on  use  and  application  of  reported  results. 
With  out  a systematic  process  for  placement  of  the  animation  the  net  effect  may  well  be  supporting  an 
instructional  objective  that  is  already  sufficiently  addressed  in  the  instruction  thereby  wasting  the  time  and 
attentional  resources  of  the  participants  as  they  would  have  responded  with  the  correct  answer  before  the  added 
stimulus  was  introduced. 
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Literature  Review  Summary 

The  numerous  studies  addressing  animation  and  its  effects  are  asking  good  questions,  but  are  for  the 
most  part  lacking  in  execution  in  some  major  areas  of  concern.  Gagne  (1985)  in  his  book  entitled  The 
Conditions  of  Learning,  proposed  that  there  are  different  types  of  learning  and  that  each  type  of  learning 
requires  a different  approach.  The  problem  remains  that  while  all  the  previously  mentioned  studies  are  asking 
good  questions  they  are  not  providing  sufficiently  based  answers.  The  need  for  the  systematic  placement  of 
independent  variables,  which  previous  studies  lack,  and  the  use  of  sound  instruments  with  reported  reliability, 
which  previous  studies  also  lack,  is  paramount  in  accurate  interpretation  of  results. 

How  This  Study  is  Different 

Where  this  study  set  itself  apart  from  the  previous  studies  in  the  literature  is  through  the  systematic 
placement  of  the  animation  in  a programmed  instructional  unit.  Further,  in  this  study  the  researchers  have  gone 
to  great  pains  in  the  pilot  studies  to  refine  the  programmed  instruction  so  that  the  participants  have  the  necessary 
factual  and  conceptual  knowledge  to  build  upon  for  higher  order  learning  to  take  place.  “An  awareness  of  the 
fact  that  that  there  are  different  kinds  of  educational  objectives  each  requiring  specific  prerequisites  is  crucial  to 
educators  who  aspire  to  employ  the  visual  media  effectively”  (Dwyer,  1978,  p.  43).  Additionally  this  study 
used  instruments  with  previously  reported  reliability  and  demonstrated  discriminatory  power  for  its  dependent 
measures.  It  is  hypothesized  that  the  use  of  the  animation  can  reduce  the  overall  cognitive  load  on  the 
participant  there  by  allowing  them  to  queue  to  the  import  information  in  the  instruction  and  process  it  more 
effectively. 


Statement  of  Purpose 

Specifically,  this  study  sought  to  examine  the  effectiveness  with  which  different  types  of  stimuli, 
varied  animation  strategies,  can  be  used  to  complement  a web-based  programmed  instruction  unit  to  improve 
learner  achievement  on  four  different  types  of  educational  objectives. 

Design  and  Methodology 

Eight-Eight  undergraduate  students  enrolled  in  lower-division  management,  educational  psychology, 
and  information  sciences  technology  classes  were  randomly  assigned  to  one  of  three  treatments  in  a randomized 
1X3  post-test  only  experimental  design.  The  type  of  animation  strategy  used  was  considered  to  be  the 
independent  variable  with  three  levels  (control  with  base  animation  employed,  simple  level  with  base  animation 
and  simple  reveal,  and  complex  level  with  base  animation  and  progressive  reveal).  Dependent  variables  were 
the  scores  achieved  on  the  criterion  measures  by  the  participants.  Participation  in  the  study  was  voluntary  and 
at  the  recruitment  sessions  the  students  were  able  to  select  their  preference  of  times  to  report  to  a list  of  labs 
were  researchers  would  be  waiting. 

Systematic  Placement  and  Development 

Two  pilot  studies  and  the  current  study  were  run  from  September  2003  through  September  2004  to 
systematically  apply  the  effective  use  of  the  animated  stimulus  in  the  final  iteration  reported  here.  A brief 
explanation  of  the  two  pilot  studies  and  treatments  development  will  be  covered  in  the  following  section. 

Pilot  Study  #1:  (n  = 12)  This  first  pilot  and  genesis 
for  the  larger  study  was  done  in  response  to  criticism  in  the 
literature  that  one  of  the  reasons  for  conflicting  results  was  that 
it  was  unreasonable  to  expect  any  achievement  differences  at 
higher  levels  of  educational  objectives  if  the  lower  levels  still 
were  not  addressed  in  sufficient  detail  as  to  achieve  an 
acceptable  score  (average  of  90%). The  first  pilot  study  was 
conducted  to  facilitate  the  development  of  the  future 
instructional  treatments  and  test  the  programmed  instructional 
units  effectiveness.  This  pilot  study  took  the  instructional 
booklet  developed  by  Dwyer  and  Lamberski  in  1977  and 
ported  the  treatment  over  to  a web-based  instructional  unit  and 
added  five  quizzes  to  make  it  into  a programmed  instructional 
unit.  There  was  no  use  of  animation  and  only  a recreation  of 


Figure  1.  Sample  Pilot  Screen  Shot 
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the  static  vis  uals  used  in  the  original  paper  based  instruction  (figure  1).  The  purpose  was  to  identify  items  with 
a high  of  difficulty  (.60  or  below  on  the  drawing  and  identification  tests)  to  determine  areas  where  the  static 
visual  instruction  might  be  complemented  by  a refined  programmed  instructional  unit  to  further  improve  student 
achievement. 

Pilot  Study  #2:  (n  = 138)  This  second  pilot  built  off  of  the  lessons  learning  in  first  pilot  on  the 
programmed  instructional  unit.  Adaptations  and  corrections  were  made  in  the  programmed  instruction  in  hopes 
of  pushing  achievement  even  higher.  Additionally  there  were  questions  to  the  effectiveness  of  our  programmed 
instructional  unit.  If  you  are  building  a structure  you  need  a good  foundation,  the  factual  and  conceptual 
information  is  that  base  for  higher  order  objectives  was  specifically  stressed  in  the  programmed  instruction. 
Simple  (base)  animations  were  developed  using  flash  and  fireworks  for  the  graphical  development.  In  this 
study  three  levels  were  established:  First  there  was  the  instructional  script  from  Dwyer  and  Lamberski  (1977), 
with  static  visuals  only  totaling  20  content  pages.  The  second  treatment  (figure  2)  was  the  programmed 
instructional  version  with  the  same  script  broken  down  into  no  more  than  2 parts  or  concepts  per  webpage  for  a 
total  of  28  content  pages  with  embedded  quizzes  after  every  5 or  six  parts  of  the  heart  were  covered  in  the  first 


Figure  2.  Pilot  #2  - Simple  Screen  Shot  Figure  3.  Pilot#2  - Complex  Screen  Shot 
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After  the  study  was  run  and  the  groups  of  students  (control  n=  47,  programmed  instruction  n = 47, 
programmed  instruction  + animation  n = 44)  data  analyzed  five  items  from  the  drawing  test,  nine  items  from  the 
terminology  test,  and  ten  items  from  the  comprehension  test  and  zero  items  on  the  identification  test  were 
identified  with  a difficultly  of  .60  or  less  and  the  animation  developed  for  the  current  study 


Current  Study:  In  this  iteration  of  the  study  there  are  three  levels  of  animations  explored.  The 
complex  treatment  from  pilot  study  number  2 with  the  placed  animation  only  at  the  right  of  center  text  area 
served  as  the  base  animation  or  control  group.  The  second  treatment  (figure  4)  used  the  same  base  animation  as 
the  control  but  added  another  effect  of  a simple  reveal  and  fade  that  cycled  two  times  than  stayed  on  the  screen 
until  the  subject  moved  on.  The  third  treatment  (figure  5)  used  the  same  base  animation  as  the  control  but 
added  another  effect  of  a progressive  reveal  and  fade  that  cycled  two  times  than  stayed  on  the  screen  until  the 
subject  moved  on.  The  belief  is  that  the  forced  information  processing  through  the  use  of  progressive  display  of 
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information  will  not  only  be  more  effective  in  directing  attention  but  also  establish  links  in  memory  that  the 
simple  and  control  treatments  may  not. 

In  the  complex  or  progressive  reveal  treatment  the  subject  was  presented  three  units  (a  word,  or  a 
symbol,  or  a group  of  words)  of  information  in  succession.  The  simple  reveal  treatment  had  the  same 
information  but  it  was  displayed  and  faded  in  all  at  once.  Additionally,  in  the  progressive  reveal  treatment  the 
part  of  the  heart  or  first  part  of  the  statement  faded  in,  then  in  the  on  screen  animation  the  corresponding  part 
blinked  three  times,  then  the  next  two  sections  of  the  phrase  or  idea  faded  in  after  a short  delay.  For  each 
thought  or  idea  the  process  was  repeated  once  for  a forced  display  of  the  information  happening  twice  on  any 
given  screen.  At  the  conclusion  of  this  animated  sequence  the  subject  could  choose  to  view  the  entire  animation 
again  or  move  on  to  the  next  webpage.  It  should  be  noted  that  the  complex  animations,  by  the  very  nature  of 
the  progressive  reveal,  required  more  time  to  display  and  averaged  around  22  seconds  run  time  for  the  base 
animation  and  the  progressive  reveal.  In  the  simple  treatments  where  the  idea  was  revealed  all  at  once  after  the 
base  animation  the  average  run  time  was  around  15  seconds.  In  the  control  group  where  only  the  base 
animation  was  displayed  the  average  was  around  11  seconds. 

The  subjects  were  given  a moderate  level  of  user  control,  they  could  replay  or  view  previous 
animations  at  any  time  during  the  instruction,  but  were  required  to  view  the  animation  on  a screen  before  being 
allowed  to  progress  through  the  lesson.  Additionally  the  environment  allowed  that  subjects  to  pull  up  static 
images  of  all  the  parts  of  human  heart  after  they  had  been  covered  in  the  instructional  unit.  There  was  a degree 
of  linearity  in  programmed  instructional  part  of  the  materials  as  there  were  required  to  correctly  answer  4 out  of 
6 questions  on  the  quizzes  or  they  were  automatically  sent  back  the  beginning  of  that  section,  generally  3 pages 
prior.  If  an  acceptable  score  was  achieved  on  the  practice  quizzes  they  were  able  to  progress  forward  normally. 

Dependent  Measures 

In  the  pilot  studies  and  in  this  iteration  of  the  study  four  criterion  measures  developed  by  Dwyer 
(1965)  each  consisting  of  20-items  was  employed  to  assess  the  participant’s  achievement.  It  should  be  noted 
that  with  the  exception  of  the  drawing  test,  which  was  administered  in  a paper  pencil  format,  all  other 
assessments  and  quizzes  were  administered  in  an  online  format  to  minimize  any  mixed  medium  effects  between 
the  lesson  and  the  assessment.  Reported  reliability  coefficients  (KR-20)  and  brief  explanation  adapted  from 
Dwyer  (1978,  pp.  45-47)  of  each  tool  are  outlined  below: 

Drawing  Test.  (K-R  20:  .91)  A 20  object  queued  recall  test  designed  to  evaluate  student  ability  to  construct 
and/or  reproduce  items  in  their  appropriate  context.  The  drawing  test  provided  the  students  with  a 
numbered  list  of  terms  corresponding  to  the  parts  of  the  heart  discussed  in  the  instructional  presentation. 

The  students  were  required  to  draw  a representative  diagram  of  the  heart  and  place  the  numbers  of  the 
listed  parts  in  their  respective  positions.  For  this  test,  the  emphasis  was  on  the  correct  positioning  of  the 
verbal  symbols  with  respect  to  one  another  and  in  respect  to  their  concrete  referents.  Conceptual  level 
educational  objectives  were  addressed  with  this  assessment. 

Identification  Test.  (K-R  20:  .86)  A 20  question  - 5 option  multiple  choice  test  designed  to  evaluate 
student  ability  to  identify  parts  or  positions  of  an  object  in  the  heart.  This  multiple -choice  test  required 
students  to  identify  the  numbered  parts  on  a supplied  detailed  drawing  of  a heart.  Each  part  of  the  heart, 
which  had  been  discussed  in  the  presentation,  was  numbered  on  the  drawing.  The  objective  of  this  test  was 
to  measure  the  ability  of  the  student  to  use  visual  cues  to  discriminate  one  structure  of  the  heart  from 
another  and  to  associate  specific  parts  of  the  heart  with  their  proper  names.  Factual  level  educational 
objectives  were  addressed  with  this  assessment. 

Terminology  Test.  (K-R  20:  .87)  A 20  question  - 5 option  multiple  choice  test  designed  to  measure 
knowledge  of  specific  facts,  terms,  and  definitions.  This  multiple -choice  test  consisted  of  items  designed  to 
measure  knowledge  of  specific  facts,  terms,  and  definitions.  The  objectives  measured  by  this  type  of  test 
are  appropriate  to  all  content  areas  that  have  an  understanding  of  the  basic  elements  as  a prerequisite  to  the 
learning  of  concepts,  rules,  and  principles.  Specifically,  conceptual  level  educational  objectives  were 
addressed  with  this  assessment  as  a spring  board  for  high  level  objectives. 

Comprehension  Test.  (K-R  20:  .84)  A 20  question  - 4 option  multiple  choice  test  designed  to  measure  a 
type  of  understanding  in  which  the  individual  can  use  the  information  being  received  to  explain  some  other 
phenomenon.  This  multiple -choice  test  consisted  of  items  where  given  the  location  of  certain  parts  of  the 
heart  at  a particular  moment  of  the  heart  beat  cycle,  the  student  was  asked  to  determine  the  position  of  other 
specified  parts  in  the  heart  at  the  same  time.  This  test  required  that  the  students  have  a thorough 
understanding  of  the  heart,  its  parts,  its  internal  functioning,  and  the  simultaneous  processes  occurring 
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during  the  systolic  and  diastolic  phases  of  the  heart  beat.  The  comprehension  test  was  designed  to  measure 
a type  of  understanding  in  which  the  individual  can  use  the  information  being  received  to  explain  some 
other  phenomenon  and  addressed  the  rule  / procedural  knowledge  educational  objectives. 

Total  Criterion  Score.  (K-R  20:  .95).  All  items  contained  in  the  previously  noted  individual  tests  were 
combined  into  a composite  test  score  to  measure  overall  learning  and  understanding. 

Validity  of  Dependent  Measures 

Upon  examination  of  the  content  pages,  instructional  script  and  materials/hce  validity  is  believed  to  be 
evident  by  the  researchers.  Given  that  the  tests  were  textually  identical  in  the  web  based  version  used  in  this 
study  content  and  construct  validity  for  dependent  measures  is  assumed  as  part  of  the  original  instruments 
umbrella.  An  explanation  of  the  process  used  in  establishing  validity  is  covered  in  detail  greater  in  Dwyer  1978 
and  in  his  1965  thesis.  In  short  the  instructional  materials  and  dependent  measures  were  put  through  content 
expert  and  educational  expert  review  with  objectives  and  descriptions  were  developed  and  employed  during  test 
development  phase  by  Dwyer  originally  in  1965  as  part  of  his  doctoral  thesis. 

Results 

For  statistical  analysis  the  alpha  level  was  set  a priori  at  the  0.05  level,  and  an  ANOVA  was  conducted 
on  each  dependent  measure.  Where  significant  differences  are  obtained,  Scheffe  or  Dunnett  C,  according  to 
Levene’s  test  of  homogeneity  of  variance,  follow  -up  procedures  were  implemented.  From  the  outset  of  the 
study  it  was  the  intention  of  the  researchers  to  analyze  the  data  from  each  criterion  measure  individually  and 
collectively  and  in  an  effort  to  comprehensively  examine  the  effects  the  animation.  In  the  table  1 below  the 
descriptive  statistics  for  all  items  are  displayed. 


Table  1.  Means  and  Standard  Deviations  for  Each  Treatment  on  Each  Dependent  Measure 


Dependent  Measure 
Mean  # Correct 
Standard  Deviation. 

Base  Animation 
Control  Group  (n=29) 

Base  Animation  with 
Reveal  (n=31) 

Base  Animation  with 
Progressive  Reveal 
(n=28) 

Drawing 

Mean 

15.97 

16.00 

18.07 

S.D. 

3.26 

3.61 

2.26 

Identification 

Mean 

17.62 

18.35 

18.45 

S.D. 

2.47 

1.70 

1.99 

Terminology 

Mean 

11.93 

13.65 

15.10 

S.D. 

5.03 

3.96 

3.30 

Comprehension 

Mean 

11.00 

12.55 

12.79 

S.D. 

3.64 

3.67 

3.51 

Total  Criterion  Score 

Mean 

56.52 

45.55 

64.75 

S.D. 

11.90 

7.76 

8.75 
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Tables  2 and  3 show  the  results  of  the  One-Way  Anova  Figure  6.  Total  Criterion  Means  Plot 
and  the  appropriate  follow  up  tests  used.  Significant  effects 
were  found  on  the  Drawing  (F=4.29  df  (2,85),  p=.02)  and 
Terminology  (F=4.25  df  (2,86),  p=.02)  and  Total  Criterion 
(F=29.767  df  (2,85),  p=.001).  In  the  case  of  the  drawing  test, 
after  using  Scheffe  post  hoc  test  it  was  determined  that  the 
progressive  reveal  group  did  significantly  better  than  both  the 
simple  reveal  and  control  groups.  For  the  Terminology  test, 
after  using  Dunnett  C follow  up  test  because  it  failed  the  Levene 
test  of  homogeneity  it  was  determined  that  the  progressive 
reveal  group  statistically  performed  significantly  better  than  the 
control  group  with  base  animation  only.  When  looking  at  the 
Total  Criterion  measure  the  assumption  of  Levene’ s test  of  1 • 

homogeneity  test  was  not  supported  and  Dunnett  C was 

implemented  on  the  results.  The  results  were  statistically  significant  in  all  comparisons  and  best  illustrated  by 
Figure  6. 


Table  2.  Results  of  Analysis  of  Variance  Across  All  Items 


Sums  of 
Squares 

df 

Mean 

Square 

F 

sig 

Drawing 

Between  groups 

83.26 

2 

41.63 

4.29 

.016 

Within  Groups 

824.82 

85 

9.70 

Total 

908.08 

87 

Identification 

Between  groups 

12.00 

2 

6.00 

1.40 

.253 

Within  Groups 

369.10 

86 

4.29 

Total 

381.10 

88 

Terminology 

Between  groups 

146.26 

2 

73.13 

4.25 

.017 

Within  Groups 

1481.65 

86 

17.23 

Total 

1627.91 

88 

Comprehension 

Between  groups 

55.20 

2 

27.60 

2.189 

.118 

Within  Groups 

1084.44 

86 

12.61 

Total 

1139.64 

88 

TOTAL 

Between  groups 

5491.29 

2 

2745.64 

29.77 

.000 

Within  Groups 

7840.17 

85 

92.234 

Total 

13331.46 

87 

Table  3.  Analysis  of  Variance  Follow  Up  Tests  With  Significance  All  Items 

Test 

Follow  Up  Used 
Levene  Statistic  Sig. 

Group  1 

Group  2 

Mean 

Difference 

Drawing 

Scheffe 

Complex 

Control 

2.11 

.13 

Simple 

2.07 

Terminology 

Dunnett  C 

Complex 

Control 

3.17 

.001 

TOTAL 

Dunnett  C 

Complex 

Simple 

8.23 

.010 

Control 

19.20 

Control 

Simple 

10.97 

Also  of  interest  was  the  analysis  run  on  just  the  items  addressed  in  the  treatments.  For  the  Drawing 
test  there  were  five  items  that  had  and  item  difficulty  less  than  .60  after  an  item  analysis.  Similarly  the 
Terminology  and  Comprehension  tests  had  nine  and  ten  items  respectively  that  were  specifically  targeted. 
Table  4 displays  the  descriptive  statistics  for  the  items  addressed.  Table  5 shows  the  results  of  the  analysis  of 
variance  and  table  six  displays  the  appropriate  follow-up  tests  and  their  significant  results.  It  should  be  noted 
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that  there  were  no  items  on  the  identification  test  with  a difficulty  level  less  than  .60  and  therefore  were  not 
addressed.  Also  the  terminology  test  was  found  significant  with  comparing  all  20  items  on  the  test  but  when 
looking  at  the  9 items  addressed  specifically  it  just  misses  the  .05  alpha  level  coming  in  with  p =.075. 


Table  4.  Means  and  Standard  Deviations  for  Individual  Items  Addressed  on  Each  Dependent  Measure 


Dependent  Measure  (#  of 
items  address) 

Mean  # Correct 
Standard  Deviation. 

Base  Animation 
Control  Group  (n=29) 

Base  Animation  with 
Reveal  (n=31) 

Base  Animation  with 
Progressive  Reveal 
(n=28) 

Drawing  (5  Items) 

Mean 

3.28 

3.81 

4.50 

S.D. 

1.60 

1.28 

0.73 

Identification  (0  Items) 

N/A 

N/A 

N/A 

Terminology  (9  Items) 

Mean 

4.14 

4.77 

5.52 

S.D. 

2.49 

2.29 

2.01 

Comprehension  (10 

Items) 

5.14 

5.06 

5.21 

Mean 

1.66 

5.21 

1.78 

S.D. 

Total  Score 

Mean 

12.55 

13.65 

15.32 

S.D. 

4.56 

3.63 

3.61 

Table  5. 

Results  of  Analysis  of  Variance  Addressed  Items 

Sums  of  Mean 

Squares  Df  Square 

Only 

F 

Sig 

Drawing 

Between  groups 

21.45 

2 

10.72 

6.62 

.002 

Within  Groups 

137.63 

85 

1.62 

Total 

159.08 

87 

Identification 

None  Targeted 

Terminology 

Between  groups 

27.64 

2 

13.82 

2.68 

.075 

Within  Groups 

444.11 

86 

5.16 

Total 

471.75 

88 

Comprehension 

Between  groups 

0.31 

2 

0.15 

0.05 

.951 

Within  Groups 

258.38 

86 

3.00 

Total 

88 

TOTAL 

Between  groups 

110.72 

2 

55.36 

3.53 

.034 

Within  Groups 

1332.38 

85 

15.68 

Total 

1443.09 

87 

Table  6 

. Analysis  of  Variance  Follow  Up  Tests  With  Significance 
Addressed  Items  Only 

Test 

Follow  Up  Used 
Levene  Statistic  Sig. 

Group  1 

Group  2 

Mean 

Difference 

Drawing 

Dunnett  C 
.004 

Complex 

Control 

1.22 

TOTAL 

Scheffe 

.246 

Complex 

Control 

2.77 

Discussion 

From  the  initial  pilot  study  run  in  the  fall  of  2003  there  has  been  marked  improvement  in  the  raw  mean 
scores  with  each  successive  iteration  beyond  simple  static  visuals,  first  using  programmed  instruction,  and  then 
adding  in  basic  animation,  and  finally  with  the  additional  animation  strategy  of  simple  reveal  vs.  progressive 
reveal.  Results  of  this  study  have  shown  some  statistically  significant  findings  for  some  educational  variable 
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levels  and  on  the  whole  but  not  for  all  educational  levels. 

The  results  of  this  study  indicate  that  the  use  of  animation  when  properly  designed  and  positioned  is  an 
important  instructional  variable  for  complementing  web-based  instruction.  However,  it  should  not  be  seen  as  a 
panacea  that  can  cure  all  the  ills  of  instruction  and  education.  Results  have  also  shown  that  the  use  of  animation 
is  not  equally  effective  for  facilitating  achievement  across  all  of  the  different  levels  of  educational  objectives. 
Even  when  subjects  have  the  perquisite  knowledge  required  to  build  upon  for  success  at  the  higher  levels 
achievement  is  not  guaranteed.  In  the  case  of  this  study  the  researcher  went  to  great  pains  to  ensure  that  lower 
level  objectives  where  addressed  in  sufficient  detail  and  we  believe  that  we  succeeded  with  the  control  group 
averaging  almost  80  and  90  percent  on  the  drawing  and  identification  tests  respectively. 

One  interesting  note  was  the  large  drop  in  the  overall  and  addressed  items  mean  scores  exhibited  by 
the  simple  reveal  group.  Possible  explanations  of  this  apparent  anomaly  vary  and  were  a frequent  topic  of 
discussion  by  the  researchers.  One  suggestion  is  that  of  a statistical  anomaly.  We  randomly  assigned  all 
subjects  to  treatments  but  there  is  still  a chance  that  it  was  a random  error.  We  are  fairly  certain  that  we  have 
avoided  common  pitfalls  in  our  research  design  but  a replication  study  is  currently  planned  for  the  spring  of 
2005  and  this  competing  explanation  can  explored  in  more  detail  then.  Another  possible  explanation  discussed 
by  the  group  was  lack  of  motivation  by  the  individuals  randomly  assigned  to  this  treatment,  but  once  again  we 
used  random  assignment  this  should  have  distributed  this  equally  across  groups.  Further,  upon  inspection  of  the 
data  and  answers  keyed  into  the  assessments  there  did  not  appear  to  be  any  obvious  patterns  or  clues  indicating 
an  obvious  lack  of  effort  by  students.  Participants  were  also  allowed  to  exit  the  study  at  anytime  if  they  so 
chose.  No  one  exercised  this  option  but  all  subjects  were  made  aware  of  it  at  the  onset  when  they  read  and 
signed  their  informed  consent  forms. 

One  possibility  that  seemed  to  resonate  with  researchers  dealt  with  the  level  of  cognitive  processing 
load,  part  of  the  underlying  theoretical  basis  for  the  study,  was  exceeded.  Even  though  the  subjects  in  the 
simple  reveal  and  progressive  viewed  the  same  content  and  all  textual  and  graphical  information  was  identical, 
the  presentation  of  information  in  the  simple  reveal  increased  the  cognitive  processing  load  beyond  their 
individual  thresholds.  It  is  suggested  that  the  subjects  could  no  longer  focus  and  direct  attention  and  that 
onslaught  of  information  that  was  taken  in  for  processing  was  quickly  disposed  of.  While  experiencing  this 
process  of  overload  the  subjects  actually  missed  the  other  bits  of  information  presented  in  all  treatments  that  the 
control  and  complex  groups  were  able  to  attune  to.  In  future  studies  qualitative  data  and  questions  asking  the 
students  to  rate  the  data  presentation  may  be  included  but  make  the  assumption  that  the  subject  is  aware  of  the 
overload,  which  may  or  may  not  be  the  case. 

Another  interesting  point  that  needs  to  be  explored  further  is  that  when  all  items  in  the  comprehension 
dependent  measure  the  analysis  of  variance  returned  an  F-ratio  of  2.19,  p=.  1 2 but  when  compared  to  items 
addressed  the  F-ratio  is  .05  and  p=.95 1 . When  viewed  in  tandem  it  appears  that  most  if  not  all  of  the  variance 
increase  was  on  the  ten  out  of  twenty  items  that  weren’t  specifically  targeted  or  addressed  by  treatments. 

Further  analysis  is  required  to  address  this  question. 

What  remains  to  be  seen  is  the  transfer  and  replication  of  these  findings  in  future  studies.  Transfer  of 
the  findings  of  this  study  can  be  logically  extended  other  domain  areas  where  information  is  organized  in  a 
hierarchal  manner.  However,  widespread  generalization  of  findings  is  a dangerous  proposition;  never-the-less, 
implications  for  practitioners  can  be  generated  so  long  as  they  are  ‘taken  with  a grain  of  salt’  and  not  assumed 
to  be  fact  but  merely  a possible  lens  from  which  to  view  a problem.  The  use  of  animation  increased  overall 
achievement  on  the  parts  of  the  learners  due  in  large  part  to  effects  on  the  factual  and  conceptual  levels.  Given 
the  cost  of  development  for  use  of  animation  this  should  be  considered  within  a contextual  frame  work  of  what 
is  it  worth  to  you?  If  similar  findings  can  be  achieved  with  well  designed  and  systematically  placed  static 
visuals  the  benefits  out  weigh  the  costs.  There  were  clear  areas  where  progressive  reveal  animation  strategies 
were  better  in  facilitating  achievement.  This  relates  to  the  clarity  of  the  relationship  between  the  way  the 
animation  was  designed  and  information  presented.  When  choosing  to  utilize  animation,  specific  attention 
needs  to  be  devoted  to  the  cognitive  load  on  the  individual.  If  items  require  too  great  a processing  time  than 
they  can  do  more  detriment  than  not  using  them  at  all  in  an  instructional  module. 
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Abstract 

This  paper  considers  four  aspects  of  online  communities.  Design,  mechanisms , architecture,  and  the 
constructed  knowledge.  We  hypothesize  that  different  designs  of  communities  drive  different  mechanisms,  which 
give  rise  to  different  architectures,  which  in  turn  result  in  different  levels  of  collaborative  knowledge 
construction.  To  test  this  chain  of  hypotheses,  we  analyzed  the  recorded  responsiveness  data  of  two  online 
communities  of  learners  having  different  designs:  a formal,  structured  team,  and  an  informal,  non-structured, 
Q&A  forum.  The  designs  are  evaluated  according  to  the  Social  Interdependence  Theory  of  Cooperative 
Learning.  Knowledge  construction  is  assessed  through  Content  Analysis.  The  architectures  are  revealed  by 
Statistical  Analysis  of  p*  Markov  Models  for  the  communities.  The  mechanisms  are  then  identified  by  matching 
the  predictions  of  Network  Emergence  Theories  with  the  observed  architectures.  The  hypotheses  are  strongly 
supported.  Our  analysis  shows  that  the  minimal -effort  hunt-for-social-capital  mechanism  controls  a major 
behavior  of  both  communities:  negative  tendency  to  respond.  Differences  in  the  goals,  interdependence  and  the 
promotive  interaction  features  of  the  designs  of  the  two  communities  lead  to  the  development  of  different 
mechanisms:  cognition  balance  and  peer  pressure  in  the  team,  but  not  in  the  forum.  Exchange  mechanism  in  the 
forum,  but  not  in  the  team.  In  addition,  the  pre-assigned  role  of  the  tutor  in  the  forum  gave  rise  to  its 
responsibility  mechanism  in  that  community,  but  not  in  team  community.  These  differences  in  the  mechanisms 
led  to  the  formation  of  different  sets  of  virtual  neighborhoods,  which  show  up  macroscopically  as  differences  in 
the  cohesion  and  the  distribution  of  response  power.  These  differences  are  associated  with  the  differences  in  the 
buildup  of  knowledge  in  the  two  communities.  The  methods  can  be  extended  to  other  relations  in  online 
communities  and  longitudinal  analysis,  and  for  real-time  monitoring  of  online  communications. 

Introduction 

Building  communities  is  recognized  as  an  essential  strategy  for  online  learning.  An  online  community 
consists  of  actors  who  develop  certain  relations  among  themselves.  For  example,  some  actors  only  read  what 
others  write;  some  respond  to  queries  posted  by  others  and  some  influence  others  to  do  something  (for  example 
to  access  a web  page),  etc.  This  simple  observation  led  us  to  adopt  a network  abstraction  to  describe  online 
communities.  A network  is  a set  of  actors  - members  of  communities,  groups,  web-pages,  countries,  genes,  etc., 
with  certain  possible  relations  between  pairs  of  actors.  The  relations  may  - or  may  not  - be  hierarchical, 
symmetrical,  binary,  or  other.  Network  abstraction  is  thus  very  flexible. 

Social  Network  Analysis  (Wasserman  and  Faust  1999)  is  a useful  tool  for  studying  relations  in  a 
network.  It  is  a collection  of  graph  analysis  methods  developed  by  researchers  to  analyze  networks  which 
consist  of  precise  mathematical  definitions  of  certain  network  structures  and  the  methods  to  calculate  them. 
Examples  of  network  structures  are  cohesiveness  and  transitivity : cohesiveness  measures  the  tendency  to  form 
groups  of  strongly  interconnected  actors;  transitivity  measures  the  tendency  to  form  transitive  triad  relations  (if  i 
relates  to  j and  j relates  to  k,  then  i necessarily  also  relates  to  k).  SNA  has  been  utilized  to  analyze  networks  in 
various  areas,  whose  actors  include  politicians  (Faust,  Willet  et  al.  2002),  the  military  (Dekker  2002), 
adolescents  (Ellen,  Dolcini  et  al.  2001),  multi-national  corporations  (Athanassiou  1999),  families  (Widmer  and 
La  Farga  1999),  and  terrorist  networks  (van  Meter  2002).  SNA  methods  were  introduced  to  online  communities 
research  in  (Garton,  Haythornthwaite  et  al.  1997).  Since  then  several  scholars  have  demonstrated  the 
applicability  of  SNA  to  specific  collaborative  learning  situations  (Haythornthwaite  1998;  Wortham  1999; 
Lipponen,  Rahikainen  et  al.  2001;  Cho,  Stefanone  et  al.  2002;  de  Laat  2002;  Martinez,  Dimitriadis  et  al.  2002; 
Reffay  and  Chanier  2002;  Aviv  2003). 

Macro-level  SNA  identifies  network  macro-structures  such  as  cohesiveness.  Micro  level  SNA  reveals 
significant  underlying  microstructures,  or  neighborhoods,  such  as  transitive  triads  (Pattison  and  Robbins  2000; 
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Pattison  and  Robbins  2002).  The  identified  neighborhoods  are  the  basis  for  revealing  theories  that  explain  their 
emergence  (Contractor,  Wasserman  et  al.  1999).  For  example,  the  theory  of  cognitive  balance  explains  the 
emergence  of  transitive  triads,  which  underlies  the  macroscopic  phenomenon  of  cohesiveness.  The  precise 
definition  of  a neighborhood  will  be  given  in  section  2. 

We  examine  online  communities  of  learners  according  to  the  constructivist  perspective  (Jonassen, 
Davidson  et  al.  1995).  Rafaeli  (Rafaeli  1988)  emphasized  that  constructive  communication  is  determined  by  its 
responsiveness.  Accordingly,  we  analyze  the  network  structures  of  the  responsiveness  relation  between  actors  in 
the  online  communities.  Previous  work  (Aviv,  Erlich  et  al.  2003)  demonstrated  that  certain  macrostructures 
(cohesion,  centrality  and  role  groups)  are  correlated  with  the  design  of  the  communities  and  with  the  quality  of 
the  constructed  shared  knowledge.  In  this  study,  we  extract  the  micro-level  neighborhoods  of  the  same 
communities.  Our  goal  is  to  reveal  the  underlying  theoretical  mechanisms  that  control  the  dynamics  of  the 
communities  and  to  correlate  them  with  the  design  parameters  and  with  the  quality  of  the  knowledge 
constructed  by  the  communities. 


Architecture  of  a Community 

An  online  community  is  modeled  as  a network  of  actors.  Every  ordered  pair  of  actors  has  a potential 
response  tie  relation.  The  response  tie  between  actor  i and  actor  j is  realized  if  i responded  to  at  least  one 
message  sent  by  j to  the  community;  otherwise  the  response  tie  is  not  realized.  In  addition,  a (non-directed) 
viewing  relation  is  realized  between  a pair  of  actors  if  they  read  the  same  messages.  In  a broadcast  community, 
a realized  response  tie  relation  is  also  a realized  viewing  tie.  The  reverse  is  not  necessarily  true. 

A virtual  neighborhood  (VN)  is  a sub-set  of  actors,  endowed  with  a set  of  prescribed  possible  response 
ties  between  them,  all  of  which  are  pair-wise  statistically  dependent.  We  identified  the  significant  VNs  of  a 
community  by  fitting  a p*  stochastic  Markov  model  (Wasserman  and  Pattison  1996;  Robins  and  Pattison  2002) 
to  the  response  tie  data.  In  this  model,  every  pair  of  response  ties  in  a VN  has  a common  actor,  which  is  why 
they  are  interdependent.  Same  topology  VNs  are  aggregated  into  a class  of  VNs.  In  the  model  every  possible 
class  is  associated  with  a strength  parameter  that  measures  the  tendency  of  the  network  to  realize  VNs  of  that 
class.  The  basic  ideas  and  the  formulas  of  the  p*  Markov  model  are  elaborated  in  (Wasserman  and  Pattison 
1996;  Robins  and  Pattison  2002).  The  model  equations  are  presented  in  the  Appendix.  Examp  les  of  Markov 
VNs  are  presented  graphically  in  Figure  1. 


Figure  1 . Virtual  neighborhoods 


In  this  research  we  consider  the  set  of  Markov  classes  of  VNs  listed  in  Table  1. 


VN  Class 

Participating  Actors  & Prescribed  Response 

link 

All  pairs:  (i?  j)or(j?  i) 

respi 

All  pairs:  (i?  j)  fixed  i 

triggi 

All  pairs:  (j?  i)  fixed  i 

mutuality 

All  pairs:  (i?  j)  and  (j?  i) 

out-stars 

All  triplets:  (i?  j)  and  (i?  k) 

in-stars 

All  triplets:  (i?  j)  and  (k?  j) 

mixed-stars 

All  triplets:  (i?  j)  and  (j?  k) 

transitivity 

All  triplets:  (i?  j)and(j?  k)and  (i?  k) 
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cyclicity 

Table  1.  Classes  ofVNs 


All  triplets:  (i?  j)and(j?  kj  and  (k?  i) 


Tendencies  to  form  VNs  of  a certain  class  are  the  result  of  the  underlying  mechanisms.  Several 
candidate  mechanisms,  postulated  by  certain  network  emergence  theories  are  briefly  described  below.  See 
(Monge  and  Contractor  2003)  for  an  extensive  survey. 

The  theory  of  social  capital  (Burt  1992;  Burt  2002)  postulates  efficient  connectivity  in  the  hunt  for  a 
social  capital  mechanism.  In  an  online  broadcast  community,  efficiency  means  forming  zero  response  ties 
because  a response  tie  is  a redundant  viewing  tie,  so  actors  prefer  to  remain  passive.  This  mechanism  predicts  a 
tendency  for  not  creating  VNs  of  any  class.  Thus,  other  mechanisms  are  responsible  for  creating  responsiveness. 

Exchange  and  resource  dependency  theories  (Homans  1958;  Wilier  1999)  postulate  an  information 
exchange  mechanism,  in  which  actors  prefer  to  forge  ties  with  potentially  “resource-promising’'  peers.  This 
mechanism  creates  tendency  for  VNs  of  class  mutuality. 

The  theory  of  generalized  exchange  (Bearman  1997)  postulates  an  information  exchange  mechanism 
via  mediators.  This  theory  then  predicts  tendencies  for  n-link  cycles,  in  particular  VNs  from  the  cyclicity  class. 

Theories  of  collective  action  (Marwell  and  Oliver  1993)  postulate  a social  pressure  mechanism  that 
induces  actors  to  contribute  to  the  goal  of  the  community  if  threshold  values  of  “pressing"  peers,  existing  ties, 
and  central  actors  are  met  (Granovetter  1983;  Valente  1996).  In  that  case,  actors  will  respond  to  several  others, 
forging  out-stars  VNs. 

Contagion  theories  (Burt  1987;  Contractor  and  Eisenberg  1990)  postulate  that  the  exposure  of  actors 
leads  to  a contagion  mechanism  that  uses  social  influence  and  imitation  to  create  groups  of  equivalent  actors 
with  similar  behaviors  (Carley  and  Kaufer  1993).  Contagion  predicts  a tendency  for  VNs  of  the  various  star 
shaped  classes. 


Theories 

Social  capital 
Collective  action 

Exchange 

Generalized  exchange 
Contagion 

Cognitive  consistency 
Uncertainty  reduction 
Exogenous  factors; 
Students 

Exogenous  factors;  Tutors 


Predicted  Tendencies 

Few  single  tie  links 
If  thresholds  met  then  respond 
to  several  others 
Tendency  to  reciprocate 
Tendency  to  respond  cyclically 
Respond  to  same  as  others 
Respond  via  several  paths 
Attract  many  responses 
No  tendencies  to  respond/trigger 


Personal  tendencies  to 
respond/  trigger 


Table  2:  Research  Hypotheses 


Hypotheses 

HI;  link  < 0 

H2;  if  thresholds  met  then  out-stars  > 0 

H3;  mutuality  > 0 
H4;  cyclicity  > 0 

H5;  out-stars>  0;  in-stars  > 0;  mixed-stars  > 0 

H6;  transitivity  >0 

H7;  in-stars  > 0 

H8;  {respi  = 0 I i ? students } 

H9;  [triggi  = 0 I i ? students} 

H10;  { respi  > 0 I i = tutor} 

Hll:  [triggi  > 0 I i = tutor} 


Theories  of  cognitive  balance  (Cartwright  and  Harary  1956;  Festinger  1957;  Harary,  Norman  et  al. 
1965)  postulate  a cognition  balance  mechanism  with  a drive  to  overcome  dissonance  and  achieve  cognition 
consistency  among  actors.  This  drive  is  implemented  by  transitivity  VNs. 

The  uncertainty  reduction  theory  (Berger  1987)  postulates  drives  in  actors  to  forge  links  with  many 
others  to  reduce  the  gap  of  the  unknown  between  themselves  and  their  environment;  this  theory  predicts  a 
tendency  to  create  in-stars  (responses  to  triggering  actors)  VNs. 

Finally,  responsibilities  of  actors  influence  their  residual  personal  tendencies  toward  response  ties.  In 
this  study,  students  did  not  have  pre -assigned  responsibilities,  predicting  that  the  students’  VNs  respx  and  triggi 
will  be  insignificant.  The  tutors’  residual  tendencies  will  be  significant,  due  to  their  roles. 

The  theories,  and  predicted  tendencies  stated  as  Research  Hypotheses,  are  presented  in  Table  2. 


The  Analysis 

We  analyzed  recorded  transcripts  of  two  online  communities  - two  communities  of  students  at  the 
Open  University  of  Israel.  These  communities  were  established  for  17  weeks  during  the  Fall  2000  semester  (19 
participants)  and  the  Spring  2002  semester  (18  participants)  as  part  of  an  academic  course  in  Business  Ethics. 
Each  community  included  one  tutor.  The  designs  of  the  activities  of  the  two  communities  were  different.  The 
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Fall  2000  community  was  designed  as  a goal-directed  collaborative  team,  whereas  the  Spring  2002  community 
was  a Q&A  forum.  Flence  we  have  labeled  the  communities  “team”  and  “forum”  respectively.  The  data  is 
available  at  http://telem.openu.ac.il/courses  (password  protected). 

The  team  community  engaged  in  a formal  debate.  Participants  registered  and  committed  to  active 
participation,  with  associated  rewards  in  place.  Students  took  the  role  of  an  "advisory  committee"  that  had  to 
advise  a company  on  how  to  handle  the  business/ethical  problem  of  cellular  phone  emissions.  The  debate  was 
scheduled  as  a 5-step  process  of  moral  decision-making,  with  predefined  goals  (Geva  2000).  A unique  feature 
of  the  team  community  was  that  the  goals  of  the  debate  were  to  reach  consensus  up  to  the  point  of  writing  a 
joint  proposal  to  an  external  agency.  The  forum  community  was  open  to  all  students  in  the  course.  Participants 
were  asked  to  raise  questions  on  issues  relating  to  the  course.  We  followed  the  social  interdependence  theory  of 
cooperative  learning  (Johnson  and  Johnson  1999)  to  characterize  the  communities  according  to  four  groups  of 
parameters:  interdependence,  promotive  interaction,  pre -assigned  roles,  and  reflection.  The  two  communities 
differ  in  most  of  the  design  parameters.  Table  3 summarizes  the  differences  between  the  designs  of  the  two 
communities. 


Parameter 

Team 

Forum 

Registration  & commitment 

Yes 

No 

Interdependence:  deliverables 

Yes 

No 

Interdependence:  tasks  & schedule 

Yes 

No 

Interdependence:  resources 

Yes 

No 

Reward  mechanism 

Yes 

No 

Interdependence:  reward 

No 

No 

Promotive  interaction:  support  & help 

Yes 

No 

Promotive  interaction:  feedback 

Yes 

No 

Promotive  interaction:  advocating  achievements 

No 

No 

Promotive  interaction:  monitoring 

Yes 

No 

Pre-assigned  roles:  tutor 

No 

Yes 

Pre-assigned  roles:  students 

No 

No 

Reflection  procedures 

No 

No 

Individual  accountability 

Yes 

No 

Social  skills 

Yes 

Yes 

Table  3:  Design  of  Communities 

Previous  analysis  (Aviv,  Erlich  et  al.  2003)  analyzed  the  constructed  knowledge  and  the  macro - 
structures  of  the  communities.  The  analysis  revealed  that  the  team  community  exhibited  high  levels  of 
constructing  knowledge,  developed  a mesh  of  interlinked  cliques,  and  that  many  participants  took  on  bridging 
and  triggering  roles  while  the  tutor  remained  on  the  side.  The  forum  community  did  not  construct  knowledge, 
cohesion  was  dull,  and  only  the  tutor  had  a special  role.  In  the  team  community,  many  students  participate  in 
many  cliques,  but  the  tutor  belongs  to  only  one  clique.  In  the  forum  community,  only  one  student  and  the  tutor 
belong  to  the  two  cliques  that  developed.  In  addition,  participants  in  the  team  comrnu  nity  shared  the  role  of 
responders  among  them,  whereas  in  the  forum  community  only  the  tutor  was  a central  responder. 

The  p*  model  of  the  team  community  has  43  classes  of  virtual  neighborhoods,  each  with  its 
explanatory  and  parameter:  18  resp j,  18  triggi,  link , mutuality , transitivity , cyclicity,  and  the  three  stars. 
Similarly,  the  model  of  the  forum  community  includes  45  classes  of  virtual  neighborhoods:  19  resp ;,  19  triggi, 
link,  mutuality,  transitivity,  cyclicity,  and  the  three  stars.  The  explanatories  count  the  number  of  virtual 
neighborhoods  that  were  completely  realized  in  the  networks.  The  strength  parameters  represent  the  tendency  to 
create  (or  not)  neighborhoods  from  the  classes. 

The  analysis  of  the  p*  model  consists  of  two  steps:  In  the  first  step  we  calculate  the  explanatories.  This  was 
performed  using  the  PREPSTAR  program  (Anderson,  Wasserman  et  al.  1999).  The  second  step  involves 
solving  the  binary  logistic  regression  (equation  A5).  The  solution  provides  an  approximate  estimate  for  the 
strength  parameters.  This  step  was  performed  with  SPSS.  Details  are  provided  in  the  Appendix.  We  configured 
the  SPSS  binary  logistic  procedure  to  work  in  forward  steps,  adding  one  class  of  virtual  neighborhoods  (i.e.,  its 
explanatory)  at  a time,  according  to  its  significance,  where  significance  was  assessed  by  the  decrements  in  the 
PLLD  (Pseudo  Log-Likelihood  Deviance).  The  analysis  stops  when  no  more  significant  explanatory  variables 
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can  be  identified. 

The  analysis  revealed  three  significant  classes  of  virtual  neighborhoods  for  the  team  community,  and 
four  significant  classes  of  virtual  neighborhoods  for  the  forum  community.  The  PLLD  estimates  of  the  strength 
parameters  are  presented  in  Table  4. 


Class 

Team 

eK 

SE 

Wald 

P 

exp(0K) 

link 

3.1 

3 

.32 

97.5 

.000 

.043 

out-star 

.18 

.06 

9.6 

.002 

1.199 

transiti 

vity 

Forum 

.31 

.06 

23.9 

.000 

1.366 

link 

2.6 

.8 

10.29 

.001 

.076 

respl& 

6.1 

.12 

26.78 

.000 

456.28 

mutuali 

ty 

6.2 

1.38 

20.61 

.002 

519.92 

in-stars  - .91 

3.2 

Table  4:  Revealed  VNs 

12.39 

.000 

.041 

In  Table  4,  0K  is  the  MPLE  (maximal  pseudo-likelihood  estimator)  for  the  strength  parameter  of  class 
K of  VNs;  SE  is  an  estimate  of  its  associated  standard  error,  exp(0K)  measures  the  increase  (or  decrease,  if  0K 
negative)  in  the  conditional  odds  of  creating  a response  tie  between  any  pair  of  participants  if  that  response  tie 
completes  a new  VN  of  class  K. 

We  tested  the  hypotheses  that  0k  = 0 by  the  Wald  parameter  (QK/SE)2  which  is  assumed  to  have  chi 
square  distribution.  Table  3 shows  that  all  these  null  hypotheses  were  rejected  with  extremely  small p values. 
The  statistical  distributions  of  the  MPLEs  and  the  Wald  parameters  are  unknown  (Robins  and  Pattison  2002),  so 
inferences  are  not  precise  in  the  pure  statistical  sense. 

Results 

Few  classes  of  VNs  are  significant:  3 in  the  Team,  4 in  the  Forum.  In  particular,  the  personal  classes 
of  VNs  of  students,  respx  and  triggx,  are  not  significant.  This  corroborates  hypotheses  H8  and  H9. 

The  relative  importance  of  the  classes  of  VNs  is  depicted  by  their  contributions  to  the  goodness  of  fit  of  the 
Markov  models.  These  are  presented  in  Figure  2. 


Figure  2.  Relative  importance  of  VNs 


Figure  2 shows  that  the  global  class  link  of  the  single  response  tie  virtual  neighborhoods  is  the  most 
significant  in  both  communities:  65%  and  72%  of  the  goodness  of  fit  are  explained  by  the  tendencies  associated 
with  this  class.  Table  5 shows  that  in  both  communities  the  strength  parameter  0 of  the  link  class  is  negative. 
This  means  that  the  major  observed  phenomenon  in  both  communities  is  a significant  tendency  for  not  creating 
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single  response  tie  neighborhoods  - the  phenomenon  of  lurking.  As  elaborated  above,  this  can  be  explained  by 
basic  self  interest  - minimizing  the  effort  required  to  forge  a response  tie  vs.  the  possible  social  capital  reward, 
given  that  every  response  tie  is  a redundant  viewing  tie.  This  observation  is  in  accordance  with  hypothesis  HI. 
Note  that  the  fact  that  response  ties  are  redundant  viewing  ties  is  a feature  of  every  broadcast  community, 
irrespective  of  the  design  of  the  community. 

By  itself,  the  negative  tendency  to  create  virtual  neighborhoods  of  class  link  will  give  rise  to  a 
community  of  non-responsive  isolates.  The  actual  responsiveness  is  formed  by  the  other  neighborhoods.  These 
neighborhoods  are  quite  different  in  the  two  communities.  The  significant  virtual  neighborhoods  in  the  team 
community  are  from  the  global  classes  transitivity  and  out-stars.  The  significant  virtual  neighborhoods  in  the 
forum  community  are  from  the  personal  class  resp j8,  and  from  the  global  classes  mutuality  and  in-stars.  In  the 
subsections  below,  we  will  consider  each  of  these  virtual  neighborhoods. 

The  goodness  of  fit  of  the  Markov  model  for  the  team  community  increases  by  30%  when  the 
transitivity  class  of  virtual  neighborhoods  is  included.  In  this  community  there  is  a positive  tendency  to  create 
transitive  virtual  neighborhoods.  This  means  that  in  the  team  community,  the  likelihood  of  setting  up  a response 
tie  from  any  actor  i to  any  other  actor  j is  enhanced  (by  1.37)  if  that  tie  comp  letes  a transitive  triangle  virtual 
neighborhood.  This  is  relative  to  the  likelihood  of  setting  up  any  other  neighborhood.  No  such  preference  exists 
in  the  forum  community.  Hypothesis  H6  - the  tendency  for  creating  virtual  neighborhoods  of  the  transitivity 
class  is  positive  - is  therefore  accepted  for  the  team  community  but  rejected  for  the  forum  community. 

The  tendency  to  create  transitive  structures  can  be  explained  by  cognitive  balance  theory.  It  seems  that 
the  design  of  the  team  community  leads  to  the  cognition  balance  mechanism,  by  which  dissonance  between 
actors  and  between  their  perceptions  of  objects  is  resolved  by  balanced  paths  of  communication.  This  can  be 
attributed  to  the  interdependence  built  into  the  design  of  the  community  and  to  the  particular  goal  which  forced 
the  participants  to  reach  consensus  during  the  online  debate  (in  order  to  submit  joint  proposals).  The  forum 
community,  on  the  other  hand,  was  a series  of  typical  Q&A  sessions.  Here  each  of  the  students  participating 
was  interested  at  a certain  point  in  time  in  a specific  issue  usually  related  to  an  assignment.  The  scope  of  the 
issue  was,  in  many  cases,  limited;  it  interested  few  students.  Other  issues,  or  even  related  concepts  not  directly 
connected  to  the  query,  were  less  important  to  the  student  who  asked  the  question,  let  alone  to  other  students. 
The  lifetime  of  each  issue  was  short  (usually  until  the  assignment  due  date).  There  was  no  drive  to  settle 
conceptual  inconsistencies  regarding  past  issues,  or  dissonance  in  perceptions  regarding  others.  Thus,  no 
cognition  balance  mechanism  was  needed  and  none  was  established. 

Introducing  the  personal  class  resp  is  to  the  model  of  the  forum  community  increases  its  goodness  of  fit 
by  21%.  This  class  includes  all  the  virtual  neighborhoods  of  single  response  ties  initiated  by  N18  - the  tutor. 
This  means  that  the  residual  tendency  of  N1 8 to  respond  - above  and  beyond  the  common  tendency  accounted 
for  by  link  - is  significant.  Specifically,  in  the  forum  community  the  odds  of  setting  up  a response  tie  (i  ? j) 
increases  (by  1,280)  if  actor  i is  N18,  the  main  responder  in  this  community.  In  contrast,  the  personal  class  of 
the  tutor's  responses  in  the  team  community,  resp i,  is  statistically  insignificant,  resp \ neighborhoods  are 
therefore  not  significant  in  the  explanation  of  the  behavior  of  the  team  community.  This  simply  means  that  the 
tutor  of  the  team  community,  PI,  showed  no  tendency  to  respond.  Hypothesis  H10-  the  tutor’s  residual 
responsiveness  is  significant  - is  accepted  for  the  forum  comrnu  nity  but  rejected  for  the  team  community. 

This  difference  is  attributed  to  the  difference  in  the  role -assignment  design  of  the  two  communities, 
which  leads  to  different  responsibility  mechanisms.  The  tutor  of  the  forum  community  was  assigned  the  job  of 
responder.  The  tutor  of  the  team  community  was  - deliberately  - not  assigned  that  role.  This  results  in  a 
difference  in  their  responsibility  mechanisms  which  leads  to  the  difference  in  their  tendency  to  create  the 
personal  class  of  virtual  neighborhoods.  A similar  observation,  mentioned  above,  is  that  none  of  the  students  in 
either  community  showed  a significant  personal  residual  tendency  to  respond,  which  supports  hypothesis  H8. 
This  again  is  attributed  to  the  fact  that  students  in  both  communities  were  not  controlled  by  responsibility 
mechanisms  because  they  were  not  assigned  any  particular  role.  Similarly,  in  both  communities  every  actor 
could  trigger  others  by  posting  a question.  No  student  was  pre -assigned  the  role  of  trigger.  This  is  reflected  in 
the  insignificance  of  the  triggj  class  of  neighborhoods  (consisting  of  a single  response  tie  towards  actor  i),  in 
agreement  with  hypothesis  H9. 

We  see  that  the  tutors  in  both  communities  had  no  significant  tendency  to  trigger  others,  contrary  to 
assumption  HI  1.  Checking  the  designs  of  the  two  communities,  we  see  that  the  tutors'  behavior  was  not 
controlled  by  responsibility  mechanisms,  but  by  other  factors.  In  the  forum  community,  the  tutor  served  only  as 
a helper  or  responder;  no  initiation  of  discussion  was  designed;  accordingly,  no  triggering  role  was  assigned  to 
the  tutor.  In  the  team  community,  discussion  was  initiated  by  the  tutor,  but  the  design  of  the  collaborative  work 
dictated  that  the  tutor  should  step  aside.  The  tutor  was  therefore  not  responsible  for  triggering  others. 
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Incorporating  the  out-stars  class  increases  the  goodness  of  fit  of  the  Markov  model  for  the  team 
community  by  5%  but  has  no  significance  for  the  forum  community.  This  means  that  in  the  team  community 
the  likelihood  of  forging  a response  tie  from  any  actor  i to  an  actor  j is  enhanced  (by  1.2)  if  the  tie  completes  an 
out-star.  No  such  tendency  is  observed  in  the  forum  community. 

The  tendency  to  create  out-stars,  that  is,  to  forge  more  than  one  response  tie  can  be  explained  by  the 
contagion  theory  (hypothesis  H5)  and  the  theory  of  collective  action  (hypothesis  H2).  The  theory  of  contagion 
predicts  tendencies  toward  both  in-stars  and  mixed-stars,  but  these  predictions  were  not  supported  by  the  data 
of  either  community.  Thus,  hypothesis  H5  was  rejected  for  both  communities.  In  general,  contagion  by 
exposure,  as  found  in  friendship  relations,  is  a time  consuming  process,  which  presumably  could  not  be 
developed  during  the  short  lifetime  of  the  communities  (one  semester). 

The  hypothesis  concerning  the  theory  of  collective  action,  H2,  was  accepted  for  the  team  community 
but  rejected  for  the  forum  community.  This  theory  assumes  the  development  of  peer  pressure,  provided  that  the 
community  parameters  of  density  and  centrality  are  above  threshold  values.  This  condition  is  fulfilled  for  the 
team  community,  but  not  for  the  forum  community,  as  can  be  seen  in  Figures  2 and  3.  In  general,  developing 
peer  pressure  is  not  trivial,  as  it  has  to  overcome  the  basic  tendency  for  lurking.  In  the  team  community, 
appropriate  initial  conditions  - commitments,  interdependence,  and  in  particular  promotive  interactions  - were 
set  up,  and  peer  pressure  was  maintained  by  the  tight  schedule  of  common  sub-goals  imposed  on  the 
community.  None  of  these  features  were  designed  into  the  forum  community,  hence  the  density  and  the  number 
of  central  actors  did  not  reach  the  thresholds  required  for  peer  pressure  to  work.  In  the  absence  of  peer  pressure, 
no  drive  for  collective  action  arouse,  which  is  the  reason  for  the  non -significance  of  the  out-stars  class  of  virtual 
neighborhoods  in  the  forum  community.  The  differences  between  the  two  communities  in  the  tendencies  for 
out-stars  is  explained  quite  well  by  the  theory  of  collective  action. 

The  mutuality  class  of  virtual  neighborhoods  accounts  for  4%  of  the  goodness  of  fit  of  the  Markov 
model  for  the  forum  community.  It  has  no  significance  for  the  team  community.  This  means  that  in  the  forum 
community  the  likelihood  of  setting  up  a response  tie  from  any  actor  i to  any  actor  j is  enhanced  (by  5,000)  if 
that  tie  closes  a mutual  tie.  (As  stated  elsewhere  in  this  paper,  the  actual  number  is  not  precise).  This  is  relative 
to  the  likelihood  of  setting  up  a tie  which  is  not  part  of  a mutual  tie.  No  such  tendency  for  mutuality 
neighborhoods  exists  in  the  team  community.  Thus,  hypothesis  H3  is  accepted  for  the  forum  community  but 
rejected  for  the  team  community. 

Hypothesis  H3  predicts  a tendency  for  mutuality  virtual  neighborhoods  on  the  basis  of  the  exchange  mechanism 
postulated  by  the  theories  of  exchange  and  resource  dependency.  Actors  select  their  partners  for  response 
according  to  their  particular  resource-promising  state.  In  the  forum  community  the  actors  prefer  to  forge 
response  ties  (if  at  all)  with  partner(s)  who  usually  respond  to  them  - which  in  this  community  is  the  tutor.  The 
tutor  is  an  a priori  resource-promising  actor  as  result  of  her  pre  -assigned  role.  This  kind  of  exchange  calculus  is 
not  developed  in  the  team  community  because  actors  in  that  community  cannot  identify  a priori  resource- 
promising  actors.  Instead,  actors  in  the  team  community  chose  another  response  policy,  governed  by  the 
cognition  balance  mechanism,  of  responding  via  transitive  triads,  as  we  saw  above. 

The  in-stars  class  of  neighborhoods  accounts  for  3%  of  the  goodness  of  fit  of  the  Markov  model  to  the 
forum  community  but  has  no  significance  in  the  team  community.  From  Table  5 we  see  that  in  the  forum 
community  in-stars  is  negative.  In  the  forum  community,  the  likelihood  of  setting  up  a response  tie  from  i to  j 
decreases  if  this  tie  complements  an  in-star  neighborhood,  that  is,  if  some  other  actor  already  has  a response  tie 
with  j.  Contagion  theory  and  the  theory  of  uncertainty  reduction  both  predict  a positive  tendency  for  in-stars 
virtual  neighborhoods.  This  prediction  is  not  fulfilled.  Hypotheses  H5  and  H7  are  rejected  for  both 
communities.  As  mentioned  above,  the  fact  that  a contagion  process  did  not  develop  can  probably  be  attributed 
to  the  short  lifetime  of  the  communities  (one  semester).  In  addition,  it  seems  that  there  was  no  need  in  either 
community  to  reduce  uncertainties  by  attracting  responses  from  several  sources:  in  the  forum  community,  the 
tutor  was  assigned  this  role;  in  the  team  community,  the  rules  of  the  game  were  clearly  explained  in  the 
document  detailing  the  design  of  the  forum. 

We  have  yet  to  understand  the  negative  tendency  toward  in-stars  virtual  neighborhoods  in  the  forum 
community.  This  negative  tendency  means  that  participants  deliberately  avoid  responding  again  to  the  same 
actor.  This  phenomenon  is  explained  by  the  theory  of  social  capital:  responding  again  to  an  actor  is  a waste  of 
energy;  it  decreases  the  structural  autonomy  of  the  responder. 

Neither  community  shows  a tendency  for  mixed-stars  or  cyclicity  classes  of  virtual  neighborhoods. 
mixed-stars  is  predicted  by  the  contagion  theory,  hypothesis  H5;  the  tendency  for  cyclicity  is  predicted  by  the 
theory  of  generalized  exchange,  hypothesis  H4.  Both  hypotheses  were  rejected  for  both  communities.  As 
mentioned  above,  it  is  plausible  that  the  contagion  mechanism  could  not  develop  during  the  short  lifetime  of  the 
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communities.  The  theory  of  generalized  exchange  relies  on  knowledge  transfer  through  intermediaries,  who 
seem  to  be  unnecessary  in  online  broadcast  communities. 

Our  findings  are  summarized  in  Table  5. 


Predicted  Hypotheses  and  Tendencies 

Results 

HI:  link  < 0 
Few  single  tie  links 

Supported  for  both  communities;  feature  of 
every  broadcast  community  independent  of 
design 

H2:  If  large  density,  centralization,  and  size,  then  out- 
stars  > 0 

Respond  to  several  others 

Supported  in  team,  but  not  in  forum;  difference 
in  meeting  threshold  conditions  due  to  built- 
in/lack  of  promotive  interactions 

H3:  mutuality  >0 

Tendency  to  reciprocate  to  resource  promising  partners 

Supported  in  forum  but  not  in  team;  difference 
in  existence/non-existence  of  a priori  resource- 
promising  actors  due  to  pre -assigned  roles 

H4:  cyclicity  > 0 

Tendency  to  respond  cyclically  to  resource-promising 
partner 

Not  supported  in  either  community,  probably 
because  there  was  no  need  for  information 
exchange  via  mediators 

H5:  out-stars  > 0;  in-stars  > 0;  mixed-stars  > 0; 
transitivity  > 0 

Respond  to  same  as  other  equivalent  actors 

Not  supported  in  either  community,  probably 
because  contagion  process  could  not  develop  in 
the  short  lifetime  of  the  communities 

H6:  transitivity  >0 
Respond  via  several  paths 

Supported  in  team,  but  not  in  forum;  difference 
due  to  difference  in  consensus  reaching 
requirements  and  interdependence 

H7:  in-stars  > 0 

Attract  responses  from  several  others 

Not  supported  in  either  community; 
uncertainties  were  clarified  by  the  design  (in 
team)  and  by  the  tutor  (in  forum) 

H8:  {respi  = 0 1 i ? students} 

H9:  { triggi  = 0 1 i ? students} 

H10:  {respi  > 0 1 i = tutor} 

HI  1:  {triggi  > 0 1 i = tutor} 

Residual  personal  tendencies  to  respond  or  trigger  only 
to  actors  with  pre  -assigned  roles 

H8,  H9:  Supported  for  both  communities;  no 
pre -assigned  role  of  responders  to  students 
H10:  Supported  in  forum,  but  not  in  team; 
differences  due  to  differences  in  pre -assigned 
roles  of  the  tutor 

HI  1:  not  supported  for  either  community;  no 
pre -assigned  role  of  triggers  to  students 

Table  5 : Summary  of  Results 

Discussion 

Our  analysis  shows  that  the  minimal-effort  hunt-for-social-capital  mechanism,  predicted  by  the  theory 
of  social  capital  & transaction  costs  controls  a large  part  of  the  behavior  of  both  communities:  a negative 
tendency  to  respond.  This  is  a feature  of  every  broadcast  community,  independent  of  design. 

Differences  in  the  goals,  interdependence,  and  the  promotive  interaction  features  of  the  designs  of  the 
two  communities  lead  to  the  development  of  different  mechanisms:  cognition  balance,  predicted  by  the  balance 
theory,  and  peer  pressure,  predicted  by  the  collective  action  theory  developed  in  the  team  community,  but  not  in 
the  forum  community.  An  exchange  mechanism  developed  in  the  forum  community,  but  not  in  the  team 
community.  In  addition,  the  unique  pre -assigned  role  of  the  tutor  in  the  forum  community  gave  rise  to  the 
responsibility  mechanism  in  that  community,  but  not  in  the  team  community.  The  differences  in  the 
mechanisms  led  to  the  formation  of  different  sets  of  virtual  neighborhoods,  which  show  up  macroscopically  as 
differences  in  cohesion  and  in  distribution  of  response  power.  These  differences  are  associated  with  the 
differences  in  the  construction  of  knowledge  in  the  two  communities  (Aviv  et  ah,  2003). 

It  should  be  noted  that  the  important  contagion  mechanism  did  not  develop  in  either  community.  This 
mechanism,  if  developed,  would  have  led  to  social  influence  and  imitation  in  attitudes,  knowledge,  and 
behavior,  which  would  have  developed  all  kinds  of  starvirtual  neighborhoods.  The  required  design  parameters 
- promotive  interaction  - were  in  place  in  the  team  community,  but  it  seems  that  the  lifetime  of  the  community 
was  too  short  for  the  development  of  this  mechanism. 

There  are  obvious  limitations  to  the  conclusions  drawn  here.  First,  we  have  considered  only  two 
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communities.  In  order  to  capture  the  commonality,  as  well  as  the  differences  in  design,  neighborhoods,  and 
mechanisms  of  online  communities,  one  needs  to  consider  a larger  set  of  communities  of  different  sizes,  topics, 
and,  in  particular,  with  different  designs.  Furthermore,  one  should  consider  a set  of  relations  embedded  in  these 
communities.  One  possibly  relevant  relation  between  actors  is  common  interest,  which  can  be  captured  by 
common  keywords  in  the  transcripts  and/or  common  sets  of  visited  web-pages. 

Another  limitation  lies  in  restricting  ourselves  to  Markov  neighborhoods.  Pattison  and  Robbins 
(Pattison  and  Robbins  2002)  emphasized  the  possible  importance  of  non-Markovian  neighborhoods  and  brought 
initial  evidence  for  the  empirical  value  of  models  that  incorporate  such  neighborhoods.  Thus,  the  dependence 
structures  can,  and  perhaps  should,  be  treated  as  a hierarchy  of  increasingly  complex  dependence  structures. 

It  seems  that  SNA  can  be  a useful  research  tool  for  revealing  community  architectures  and  mechanisms 
of  online  communities.  There  are  numerous  directions  for  future  research.  One  direction  is  “community- 
covariates  interaction.”  Several  studies,  such  as  (Lipponen,  Rahikainen  et  al.  2001),  revealed  that  certain 
participants  take  on  the  roles  of  influences  (who  trigger  responses)  or  of  celebrities  (who  attract  responses). 
Others  are  isolated  - no-one  responds  to  them  or  is  triggered  by  them.  The  question  is  whether  this  behavior 
depends  on  individual  attributes  or  whether  this  is  universal  and  found  across  communities.  Another  direction  is 
“community  dynamics,”  an  inquiry  into  the  time  development  of  community  structures.  When  do  cliques 
develop?  Are  they  stable?  What  network  structures  determine  this  behavior?  Yet  another  direction  is  “large 
group  information  overload.”  It  is  well  known  that  the  dynamics  of  large  groups  leads  to  boundary  effects  that 
occur  when  the  group  and/or  the  thread  size  increases  (Jones,  Ravid  et  al.  2002).  How  are  these  manifested  in 
online  communities? 

One  practical  implication  of  the  methodology  used  here  is  the  possibility  of  online  monitoring  and 
evaluation  of  online  communities,  by  embedding  SNA  tools  into  community  support  environments.  But  a word 
of  caution  is  necessary:  There  are  various  definitions  of  network  structures.  Experience  shows  that  applying 
different  definitions  may  lead  to  different,  even  contradictory,  results.  Further  research  is  needed  to  determine 
the  stability  of  network  structures  under  such  redefinitions. 

Appendix:  Key  Ideas  of  the  p*  Markov  Model  and  the  Estimation  Procedure 

In  this  research  we  construct  parameterized  p*  Markov  models  for  the  two  networks,  assuming 
isomorphism  invariance,  thereafter  extracting  the  parameters  via  the  MPLE  (maximum  pseudo-likelihood 
estimation)  procedure  (Strauss  and  Ikeda  1990).  Details  of  the  precise  formulation  of  the  models,  assumptions 
and  the  analysis  are  presented  in  a series  of  papers  (Wasserman  and  Pattison  1996;  Anderson,  Wasserman  et  al. 
1999;  Pattison  and  Robbins  2002).  In  this  section  we  present  the  key  ideas  required  for  understanding  the  results 
and  their  interpretation. 

Any  ordered  pair  of  actors  in  a network  has  a potential  response  tie  relation.  We  say  that  the  response 
tie  relation  between  actor  i and  actor  j is  in  the  realized  state  if  i responded  to  at  least  one  message  sent  by  j to 
the  network.  Otherwise  a response  tie  is  in  the  un-realized  state.  The  state  of  the  network  of  g actors  is  then 
defined  by  the  gXg  response  matrix  r : ry  = 1 if  a response  tie  between  i and  j is  realized,  otherwise  r,j  = 0.  The 
states  of  the  response  ties  are  assumed  to  be  the  result  of  stochastic  mechanisms  operating  between  pairs  of 
actors.  Furthermore,  we  assume  that  the  probability  that  the  response  matrix  will  actually  develop  into  a state  r, 
Pr(r),  is  an  exponential  function  of  a linear  combination  of  p state  dependent  explanatory  variables  or 
explanatories,  { zi (r),  Z2(r),  ...,  Zp(r)}.  Each  explanatory  zi(r)  has  an  associated  unknown  strength  parameter  0j. 
Estimates  of  the  strength  parameters  are  obtained  by  fitting  the  observed  states  of  the  response-ties  in  the  forum 
to  the  predictions  of  the  probability  function  P(r). 


Pr(r)  = exp{9|  Zi(r)  + 02  z2(r)  + ...  + 0p  Zp(r)}  / K(0j,  02,  ...,  0p) 


(Al) 


The  Hamersley-Clifford  theorem  (Besag,  1974,  1975)  states  that  each  explanatory  ZN(r)  and  its 
associated  strength  parameter  0n  are  associated  with  one  virtual  neighborhood  N.  A virtual  neighborhood  is  a 
sub-set  of  actors  and  prescribed  possible  response  ties  between  them,  all  of  which  are  pair-wise  statistically 
dependent.  Actors  in  a neighborhood  may  be  physically  far  apart  (which  is  why  we  call  it  virtual),  but  due  to 
certain  mechanisms,  their  possible  response  ties  are  all  statistically  interdependent.  Note  that  the 
interdependency  of  the  prescribed  possible  response  ties  is  an  inherent  property  of  the  virtual  neighborhood 
which  in  principle  is  unrelated  to  the  actual  realization  states  of  the  response  ties.  A virtual  neighborhood  may 
be  completely  or  partially  realized,  or  not  realized  at  all.  According  to  this  definition,  two  possible  response  ties 
between  pairs  of  actors  in  different  virtual  neighborhoods  are  statistically  independent.  The  Hamersley-Clifford 
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theorem  states  that  each  virtual  neighborhood  is  associated  with  one  explanatory  and  its  strength  parameter.  The 
explanatory  measures  whether  the  virtual  neighborhood  is  completely  realized,  in  which  case  it  is  1.  Otherwise 
it  is  zero.  The  strength  parameter  quantifies  the  probabilistic  tendency  to  realize  the  virtual  neighborhood 
completely. 

A subset  of  a virtual  neighborhood  is  also  a virtual  neighborhood.  Any  single  pair  of  actors  with  a 
single  prescribed  possible  response  tie  between  them  is  by  definition  a virtual  neighborhood.  Different  virtual 
neighborhoods  might  have  the  same  set  of  actors  but  different  prescribed  response  ties.  Holland  and  Leinhardt 
(Holland  and  Leinhardt  1981)  considered  models  in  which  virtual  neighborhoods  included  only  dyads  of  actors 
with  mutual  ties.  This  implies  that  dyads  are  independent,  which  is  an  oversimplification.  Markov  models 
incorporate  a larger  variety  of  various  sizes  of  virtual  neighborhoods.  In  a Markov  neighborhood,  every  two 
prescribed  response  ties  have  one  actor  in  common  (which  is  why  they  are  dependent).  Examples  of  Markov 
virtual  neighborhoods  are  graphically  presented  in  Figure  1.  Markov  dependency  was  introduced  by  Frank  and 
Strauss  (Frank  and  Strauss  1986).  It  is  a natural  assumption  in  an  online  community:  Forging  response  ties  is  an 
effort,  so  an  actor’s  response  ties  are  conceivably  interdependent. 

The  isomorphism  invariance  (or  homogeneity)  approximation  aggregates  same -topology  virtual 
neighborhoods  into  isomorphism  classes,  each  having  one  common  strength  parameter  and  one  explanatory. 

The  explanatory  is  then  a simple  counter:  It  counts  the  number  of  virtual  neighborhoods  of  the  particular  class 
that  are  realized  in  the  network.  For  example,  the  explanatory  associated  with  the  class  of  transitive  triads 
counts  the  number  of  such  triads  that  are  realized  in  the  network.  The  strength  parameter  quantifies  the 
probabilistic  tendency  of  the  network  for  realizing  virtual  neighborhoods  of  the  class.  In  this  research  we 
consider  the  set  of  Markov  isomorphism  classes  listed  in  Table  Al.  The  three  left  -hand  columns  in  the  table 
define  the  membership  of  actors  in  each  class  and  the  prescribed  possible  response  ties,  the  name  of  the 
associated  strength  parameter  (which  also  serves  as  the  name  of  the  class  itself),  and  the  formula  for  deriving 
the  explanatory  variables  (counters)  from  the  response  matrix  r. 

Table  Al:  Classes  of  Virtual  Neighborhoods  and  Explanatories  used  in  Study 


Isomorphism  Class  of  Virtual 
Neighborhoods: 

Participating  Actors  & Prescribed 
Response  ties 

Strength 
Parameter  0 

Explanatory  zn(r) 
(counter) 

Effects:  If  0 > 0 is 
significant  ? 
enhanced  tendency  to 
create 

All  pairs  {i,j}l(i?  j)or(j?  i) 

link 

L(r)?  ? ;?  iPi 

links  (either  direction) 

All  pairs  {i,  j }l  (i?  j)  fixed  i 

respi 

Ri(r)  = ? ihi 

responses 

All  pairs  { j,  i } 1 (j?  i)  fixed  i 

trigg i 

Ti(r)=  ? irii 

triggers 

All  pairs  {i,j(l(i?  j)  AND  (j?  i) 

mutuality 

M(r)  ? ? i?  iiiira 

mutual  responses 

All  triplets  {i,  j,  k } 1 (i?  j)  AND  (i?  k) 

out-stars 

OS2(r)  ? ?? ;?  i?  krljrik 

star-responses 

All  triplets  { i,  j , k } 1 (i?  j)  AND  (k?  j) 

in-stars 

IS2(r)  ? 7?  j?  ,?  kPAj 

star-triggers 

All  triplets  {i,  j,  k}l  (i?  j)  AND  (j?  k) 

mixed-stars 

MS2(r)  ? ? ;?  j?  kri,r|k 

mixed  trigger- 
responses 

All  triplets  {i,  j,  k}l  (i?  j)  AND  (j?  k) 
AND  (i?  k) 

transitivity 

TRT(r) 

? ? i?  i?  kr  :i\rik 

transitive  triads 

All  triplets  {i,  j,  k}l  (i?  j)  AND  (j?  k) 
AND  (k?  i) 

cyclicity 

CYT(r) 

? ? i?  i?  kriirikrki 

cyclic  triads 

The  probability  function  then  takes  the  following  form: 

Pr(r)  = exp{0'  • z(r)}/ k(0?)  (A2) 

Where  the  vector  of  explanatories  consists  of  the  counters  listed  in  Table  Al: 

z(r)  = (L(r),  Ri(r),  Ti(r),  M(r),  OS2(r),  IS2(r),  MS2(r),  TRT(r),  CYT(r)}  (A3) 

and  the  strength  parameters  measure  the  tendencies  for  realizing  the  virtual  neighborhoods  of  the  corresponding 
Markov  classes: 
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0 = {link,  respi,  triggi,  mutuality,  out-stars,  in-stars,  mixed-stars,  transitivity,  cyclicity } (A4) 

The  L(r)  explanatory  counts  the  number  of  neighborhoods  of  class  link  that  were  actually  realized  in 
the  network  whose  response  matrix  is  r.  Its  strength  parameter  link  measures  the  common  tendency  to  form 
single  response  ties;  that  is,  to  respond  or  to  trigger.  If  link  is  negative,  it  measures  the  tendency  not  to  form 
response  ties. 

The  Rj(r)  explanatory  counts  the  number  of  neighborhoods  of  the  respx  class  that  were  realized  in  the 
network,  and  its  strength  parameter.  respx  measures  the  residual  tendency  (or  non-tendency)  of  actor  i to 
respond,  above  and  beyond  the  common  tendency  measured  by  link . Similarly,  Ti(r)  counts  the  number  of 
neighborhoods  of  class  triggi  that  were  actually  realized,  and  the  triggi  strength  parameter  measures  the  residual 
capability  of  actor  i to  attract  responses  to  his/her  previous  messages;  that  is,  to  trigger  others  to  respond,  above 
and  beyond  the  common  capability  measured  by  link.  The  M(r)  explanatory  counts  the  number  of  realized 
mutual  dyads.  The  strength  parameter  mutuality  measures  the  global  tendency  (or  non-tendency)  of  a network  to 
form  such  dyads.  The  OS2(r),  IS2(r),  and  MS2(r)  variables  count  the  number  of  realized  star  virtual 
neighborhoods  of  the  three  global  classes  (see  Fig.  3).  The  corresponding  strength  parameters  measure  the 
tendency  (or  non-tendency)  to  forge  response  ties  with  (and/or  fro  m)  two  partners.  The  transitivity  and  cyclicity 
global  classes  include  all  triad  virtual  neighborhoods  that  are  transitive  or  cyclic,  respectively.  The  associated 
explanatories,  TRT(r)  and  CYT(r)  count  the  number  of  virtual  neighborhoods  from  these  classes  that  were 
actually  realized,  and  the  corresponding  strength  parameters  - transitivity  and  cyclicity  - measure  the  tendency 
to  realize  virtual  neighborhoods  of  these  classes. 

It  should  be  emphasized  that  the  explanatories  count  only  completely  realized  neighborhoods:  a virtual 
neighborhood  must  have  all  its  prescribed  response  ties  realized  in  order  to  be  counted. 

Wasserman  and  Pattison  (Wasserman  and  Pattison  1996)  reformulated  the  exponential  form  of  Pr(r) 
into  a logit  form,  which  provides  both  an  insight  into  the  precise  meaning  of  "tendency"  and  a useful  procedure 
for  estimating  the  strength  parameters.  The  logit  form  of  the  Markov  model  is  presented  in  equation  A5: 

wy  = log  [ Pr(rij  = II  rcjj)  / Pr/r,,  = 01  r'y)  ] = ? N0NdN(rCij,  ij)  (A5) 

The  left  hand  side  is  the  logit  - the  log  of  the  conditional  odds  of  a pair  of  actors  (i,  j)  to  realize  a 
response  tie  (i  ? j).  Here  the  odds  (the  ratio  between  the  probability  for  realizing  and  not  realizing  a response 
tie)  is  conditioned  on  all  other  response  tie  states,  denoted  by  rCjj,  held  fixed.  The  logit  wy  is  a linear 
combination  of  the  changes  in  the  values  of  the  explanatories  when  the  response  tie  (i  ? j)  jumps  from  a not 
realized  to  a realized  state,  when  all  other  response  ties,  rcy  are  held  fixed: 

dN(rCij,  ij)  = zN(rCij,  r,,  = 1)  - zN(rCij,  ry  = 0)  (A6) 

The  change  statistic  dN(rcy,  ij)  counts  the  increase  in  the  number  of  virtual  neighborhoods  of  class  N 
when  the  response  tie  (i  ? j)  flips  from  "non-realized"  to  "realized."  It  is  1 if  (i  ? j)  completes  a whole  virtual 
neighborhood;  otherwise  it  is  zero. 

The  logit  form  (A5)  provides  a simple  interpretation  of  the  strength  parameters.  Suppose  that  an 
explanatory  ZN(r)  with  strength  parameter  0 is  significant.  If  this  happens  then  the  conditional  odds  for  the 
realization  of  the  response  tie  (i  ? j)  from  any  actor  i to  any  actor  j will  be  enhanced  by  e0  if  this  envisaged 
response  tie  will  make  a new  virtual  neighborhood  of  class  N realized  completely.  This  will  happen  if  the 
network  already  has  an  almost  complete  realization  of  the  neighborhood:  only  (i  ? j)  is  missing.  Otherwise  the 
conditional  odds  do  not  change.  Note  that  if  the  strength  parameter  0 is  negative,  the  conditional  odds  will  be 
decreased.  This  means  that  the  network  has  the  opposite  tendency.  For  example,  if  the  transitivity  explanatory  is 
significant,  then  the  conditional  odds  for  forming  a response  tie  (i  ? j)  is  multiplied  by  e‘ranu"v“t  pf  the  response 
tie  (i  ? j)  completes  a transitive  triad  (i  responds  to  j,  AND  j responds  to  k AND  i also  responds  to  k).  This  will 
be  larger  or  smaller  than  1 depending  on  the  transitivity  sign. 

The  logit  form  (A5)  is  the  basis  for  one  method  of  estimating  the  strength -parameters.  This  method - 
the  MPLE  procedure  - treats  A5  as  a binary  logistic  regression  equation:  the  response  tie  variable  is  the 
dependent  variable.  There  are  g(g-l)  cases:  each  ordered  pair  of  actors  (i,  j)  is  one  case.  The  values  of  ry  (1  or  0) 
for  all  cases  are  the  observed  response  ties.  The  independent  variables  in  the  regression  equation  are  the 
“change  statistics”  dN(rcy,  ij)  associated  with  the  explanatories.  The  coefficients  of  the  change  s tatistics  in  the 
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regression  equation,  0n,  are  the  unknown  strength  parameters  of  the  corresponding  explanatories. 

To  solve  A5  and  estimate  the  strength  parameters,  one  constructs  the  pseudo  log  likelihood  function: 

PL(0)  = ? ijlog  [ Prfej  - II  rc;j)  / Pr(rij  = 01  rcy)  ] = ? y?  N0NdN(rCij,  ij)  (A7) 

PL(0)  is  the  log  of  the  product  of  all  the  conditional  probabilities.  It  is  considered  a function  of  the 
unknown  strength  parameters  0?  ? 04L  ?...0p?  Avith  the  response  tie  states  r fixed  at  the  observed  values.  The 
estimators  of  the  strength  parameters  are  then  the  values  of  0i,  02, ...  0p  that  maximize  PL(0).  These  are  the 
Maximum  Pseudo  Likelihood  Estimators  (MPLEs).  The  problem  with  this  method  is  that  the  statistical 
distributions  of  these  estimators  are  not  known.  One  cannot  assume  that  they  have  the  same  statistical  (chi 
squared)  distributions  as  their  MLE  (maximum  likelihood  estimator)  counterparts.  Significance  intervals  based 
on  this  assumption  can  at  best  be  considered  defendable  approximations,  not  precise  statistical  statements.  This 
study  attempts  to  identify  the  relative  strength  of  the  most  important  explanatories,  with  no  claim  to  provide 
precise  numerical  values  for  the  actual  values  of  their  strength  parameters. 

In  this  research  the  actual  values  of  the  change-statistics  dN(rcy,  ij)  were  calculated  from  the  observed 
response  r matrix  using  PREPSTAR  (Anderson  et  al.,  1999).  The  MPLEs  for  the  strength  parameters  were  then 
obtained  by  solving  equation  (A7)  using  the  binary  logistic  procedure  of  SPSS.  See  (Crouch  and  Wasserman 
1998;  Contractor,  Wasserman  et  al.  1999)  for  examples  and  details. 

Note  that  once  we  have  estimates  for  the  strength  parameters,  we  can  estimate  the  value  of  the  pseudo 
log  likelihood  function,  PL(0),  itself.  This  value,  to  be  precise  -2*  PL(0),  can  serve  as  an  estimate  for  the 
goodness  of  fit  of  the  model.  The  best  fit  is  when  the  product  of  the  conditional  probabilities  is  1,  so  that  -2* 
PL(0)  is  zero.  In  practice,  this  is  a positive  number  called  Pseudo  Log  Likelihood  Deviance  (PLLD)  signifying 
that  the  model  is  not  perfect.  What  we  are  interested  in,  however,  are  the  decrements  in  the  PLLD  caused  by 
introducing  more  explanatories  into  the  model.  A decrement,  denoted  by  ? ■> , measures  the  contribution  of  the 
virtual  neighborhood  N to  the  goodness  of  fit  of  the  model.  If  one  can  conjecture  that  PLLD  and  ? ? have  chi 
square  distributions  (as  do  their  counterparts  in  MLE  procedure),  then  one  has  precise  numerical  estimates  for 
the  relative  importance  of  their  contributions  to  the  goodness  of  fit.  As  stated  above,  this  assumption  is 
approximate  at  best.  Therefore  the  decrements  ? ? serve  as  a guide  to  the  relative  importance  of  the  virtual 
neighborhoods,  but  do  not  provide  the  range  of  the  true  values. 
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Abstract 

This  study  explored  teacher  educators ' pedagogical  beliefs  and  technology  uses  in  relation  to 
presen’ice  teachers  ’ pedagogical  beliefs  and  attitudes  toward  technology.  Correlation  and  regression  analysis 
were  conducted  to  answer  the  research  questions.  The  results  revealed  some  relationships  between  the  teacher 
educators  ’ beliefs  and  their  uses  of  technology.  In  addition , it  was  found  that  the  teacher  educators  ’ learner- 
centered  beliefs  could  influence  the  presen’ice  teachers’  learner-centered  beliefs.  The  frequency  that  the 
teacher  educators  had  the  presen’ice  teachers  use  technology  in  both  constructivist  and  traditional  way  could 
influence  the  presen’ice  teachers ' attitudes  toward  technology. 

Introduction 

The  rapid  development  of  information  technology  has  made  computers  and  computer-related 
technology  an  integral  part  of  teaching  and  learning.  According  to  Glenn  (1997),  computers  have  advanced 
from  simple  machines  with  limited  functions  and  capabilities  to  powerful  machines  with  sophisticated 
applications  and  high-speed  networking  capabilities.  Since  the  mid-1970s,  schools  districts  have  raced  to  keep 
up  with  the  rapid  growth  and  change  of  technologies.  Under  such  conditions,  it  is  necessary  for  teachers  to  learn 
new  pedagogical  and  technological  skills  to  better  facilitate  students’  learning  in  classrooms  (Glenn,  1997). 

To  better  prepare  teachers  to  integrate  technology  in  their  K- 12  classrooms,  teacher  educators  should 
take  the  responsibility  to  prepare  future  teachers  by  infusing  technology  in  their  education  courses  (Vannatta  & 
O’Bannon,  2002;  Willis  & Tucker,  2001).  Faculty  members  should  be  prepared  to  model,  support,  and  require 
technology  use  by  students  (Cuban,  1995).  Ertmer  (1999)  described  two  barriers  to  technology  integration:  first- 
order  barriers  and  second -order  barriers.  First-order  barriers  are  extrinsic  to  teachers  and  include  lack  of  access 
to  hardware  and  software,  time,  and  necessary  support.  Second-order  barriers  are  intrinsic  to  teachers,  including 
teachers’  belief  systems  about  teaching  and  learning  and  practices  in  teaching.  Since  the  second-order  barriers 
are  more  ingrained  and  personal,  they  may  cause  more  difficulties  to  overcome.  Furthermore,  second-order 
barriers  can  affect  meaningful  technology  integration.  Therefore,  as  a second-order  barrier,  teachers’ 
pedagogical  beliefs  may  play  an  important  role  in  the  ways  in  which  technology  gets  used  in  classrooms. 

Richardson  (1996)  defined  beliefs  as  “psychologically  held  understandings,  premises,  or  propositions 
about  the  world  that  are  felt  to  be  true”  (p.  103).  Some  previous  studies  (Becker,  1999;  Bigatel,  2002; 
Niederhauser  & Stoddart,  2001)  have  suggested  that  inservice  teachers’  beliefs  about  teaching  and  learning  had 
impact  on  their  uses  of  technology  in  the  classroom.  Compared  to  the  teachers  who  had  traditional  beliefs  about 
teaching  and  learning,  the  teachers  who  had  constructivist  beliefs  were  strong  computer  users,  they  used 
computers  frequently  and  powerfully  in  their  teaching.  Instead  of  emphasizing  the  impact  that  teachers’  beliefs 
on  their  uses  of  technology,  some  researchers  (Hadley  & Sheingold,  1993,  Woodrow,  et  al.,  1996)  found  that 
technology  can  influence  teachers’  beliefs  and  attitudes.  Many  of  these  teachers  incorporated  technology  into 
their  teaching  practices,  which  deeply  affected  their  teaching  and  the  students’  learning.  Regardless  the  report 
on  the  relationship  between  inservice  teachers’  pedagogical  beliefs  and  their  uses  of  technology,  little  is  known 
about  the  relationship  between  teacher  educators’  pedagogical  beliefs  and  their  technology  uses.  The 
examination  of  such  relationship  in  this  study  is  an  exploratory  effort  to  fill  the  gap  in  literature  and  contribute 
to  our  growing  knowledge  about  faculty  development  in  technology  use  in  teacher  preparation  programs.  Such 
exploration  is  also  important  in  the  efforts  to  prepare  preservice  teachers  to  effectively  use  technology  in  their 
future  teaching. 

Teacher  educators  shoulder  the  responsibility  for  educating  technology-using  preservice  teachers. 

Since  teachers’  beliefs  exert  a powerful  influence  on  teachers’  instructional  decisions  and  classroom  practices 
(Pajares,  1992),  it  is  reasonable  to  expect  that  teacher  educators  who  have  different  pedagogical  beliefs  will 
deliver  instruction  in  different  ways,  which  in  turn,  may  have  differential  influences  over  preservice  teachers’ 
beliefs  about  teaching  and  learning.  Therefore,  the  exploration  of  teacher  educators’  pedagogical  beliefs  in 
relationship  to  preservice  teachers’  pedagogical  beliefs  is  necessary  educational  inquiry  in  teacher  education 
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programs.  It  will  help  us  learn  about  the  influence  that  teacher  educators  may  have  on  preservice  teachers’ 
beliefs  and  add  to  our  knowledge  about  how  to  broaden  preservice  teachers’  pedagogical  beliefs  to  encompass 
beliefs  about  meaningful  uses  of  technology.  The  introduction  of  computers  and  related  technologies  into 
schools  makes  it  necessary  for  teachers  to  take  advantage  of  technology  in  instruction.  Pre  service  teachers  need 
to  be  well  prepared  in  using  technology  in  teacher  education  programs.  Beliefs  about  teaching  and  learning  play 
an  important  role  in  transforming  classrooms  with  the  use  of  technology  (Ertmer,  1999).  Understanding  how 
preservice  teachers’  pedagogical  beliefs  relate  to  teacher  educators’  pedagogical  beliefs  may  help  to  predict 
their  technology  uses  in  future  teaching. 

In  addition  to  examining  preservice  teachers’  beliefs  about  teaching  and  learning,  it  is  also  important  to 
consider  preservice  teachers’  attitudes  toward  technology  use  in  education.  According  to  Aiken  (1980),  attitudes 
“may  be  conceptualized  as  learned  predispositions  to  respond  positively  or  negatively  to  certain  objects, 
situation,  concepts,  or  persons’’  (p.  2). 

Attitudes  had  influence  on  teachers’  uses  of  technology  in  classrooms  (Boone  & Gabel,  1994;  Levine 
& Donitsa-Schmidt,  1998;  Piper,  2003).  Being  familiar  with  technology  does  not  necessarily  mean  preservice 
teachers  perceive  that  technology  has  a use  in  the  classroom,  and  therefore,  they  may  not  be  willing  to  teach 
with  technology  in  the  classroom  (Ropp,  1999).  Attitudes  toward  technology  did  not  only  influence  the 
student’s  initial  adoption  of  computer  technology,  but  also  their  future  uses  (Selwyn,  1997).  Teachers’  positive 
attitudes  toward  technology  will  make  them  likely  to  use  it  in  the  future  (Yildirim,  2000).  Thus,  it  is  important 
to  understand  the  factors  related  to  preservice  teachers’  attitudes  toward  technology. 

Some  researchers  (Abbott  & Faris,  2000;  Kumar  & Kumar,  2003)  have  suggested  that  preservice 
teachers’  attitudes  toward  technology  could  be  improved  by  integrating  technology  into  teacher  education 
course  work.  However,  few  studies  have  been  conducted  to  directly  connect  teacher  educators’  uses  of 
technology  and  preservice  teachers’  attitudes  toward  technology.  “Teachers  teach  as  they  have  been  taught,  and 
it  is  unlikely  that  computer  skills  will  be  transferred  to  students  and  encouraged  by  teachers  unless  the  teachers 
have  positive  attitudes  toward  computer  use’’  (Yildirim,  2000,  p.  481).  Thus,  it  is  necessary  to  explore  how 
teacher  educators,  as  models  of  teaching  and  technology  use  in  classrooms,  influence  preservice  teachers’ 
attitudes  toward  technology.  Aiken  (1980)  used  modeling  theory  to  analyze  the  development  and  the  change  of 
attitudes. 

Many  attitudes  are  not  the  result  of  direct  reinforcement  but  are  learned  by  observing  the  activities  of 
people  who  are  perceived  as  significant.  As  a person  grows  to  maturity,  numerous  individuals - 
parents,  peers,  and  television  stars,  among  others  - serve  as  models  of  attitudes  and  behavior.  In  the 
process  of  modeling  the  behavior  of  people  who  are  important  to  her  or  him,  a person  makes 
provisional  attempts  to  act  and  believe  as  the  model  is  perceived  to  act  and  believe  (p.  16). 

Therefore,  it  is  a worthwhile  effort  to  examine  the  relationship  between  teacher  educators’  technology 
uses  in  instruction  and  preservice  teachers’  attitudes  toward  technology.  This  effort  is  important  in  identifying 
the  factors  that  have  influence  on  preservice  teachers’  attitudes  toward  technology  and  facilitating  preservice 
teachers’  positive  attitudes  toward  using  technology  in  their  future  teaching. 

Therefore,  the  purpose  of  this  study  was  to  examine  how  teacher  educators’  underlying  beliefs  about 
teaching  and  learning,  or  pedagogical  beliefs,  were  related  to  their  uses  of  technology  in  instruction.  Also,  this 
study  intended  to  explore  the  relationship  between  teacher  educators’  pedagogical  beliefs  and  preservice 
teachers’  pedagogical  beliefs,  as  well  as  the  relationship  between  teacher  educators’  uses  of  technology  and 
preservice  teachers’  attitudes  toward  technology  uses  in  classrooms. 

Specifically,  this  study  will  answer  the  following  questions: 

1.  What  is  the  relationship  between  teacher  educators’  pedagogical  beliefs  and  their  technology  uses? 

2.  What  is  the  relationship  between  teacher  educators’  pedagogical  beliefs  and  preservice  teachers’ 
pedagogical  beliefs? 

3.  What  is  the  relationship  between  teacher  educators’  technology  uses  and  preservice  teachers’  attitudes 
toward  technology? 


Methods 

Overview  of  Design 

This  study  employed  a correlational  research  design.  To  answer  the  first  question  on  the  relationship 
between  teacher  educators’  beliefs  and  their  technology  uses,  bivariate  correlational  study  method  was  used.  To 
answer  the  second  question  on  the  relationship  between  teacher  educators’  beliefs  and  preservice  teachers’ 
beliefs,  prediction  study  method  was  employed  by  using  multiple  regression  technique.  To  answer  the  third 
research  question  on  the  relationship  between  teacher  educators’  uses  of  technology  and  preservice  teachers’ 
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attitudes  toward  technology,  the  same  study  method  used  to  answer  the  second  question  was  applied. 

Participants  and  Site 

This  study  was  conducted  with  the  instructors  and  the  students  in  School  of  Education  at  a large  mid  - 
western  university  in  a spring  semester.  Convenience  sampling  method  was  used.  The  preservice  teacher 
participants  were  100  students  who  enrolled  in  two  beginning  teacher  education  courses,  course  A and  course  B. 
Students  in  these  two  courses  formed  a cohort,  which  meant  all  those  who  attended  one  section  of  course  A also 
took  the  corresponding  section  of  course  B.  Of  these  100  students,  59  of  them  were  also  taking  an  introductory 
educational  technology  course,  course  C.  A total  of  24  teacher  educators  took  part  in  this  study,  18  of  them  were 
graduate  instructors  and  6 of  them  were  faculty  members.  Of  the  24  instructors,  7 instructors  were  teaching 
course  A,  9 were  teaching  course  B,  and  1 was  teaching  course  C. 

Variables  and  Instruments 

In  general,  a total  of  four  variables  were  examined  to  answer  the  research  questions,  including  1) 
teacher  educators’  pedagogical  beliefs,  2)  preservice  teachers’  pedagogical  beliefs,  3)  teacher  educators’  uses  of 
technology  in  instruction,  and  4)  preservice  teachers’  attitudes  toward  technology  use  in  instruction. 

Teacher  educators’  beliefs  and  preservice  teachers’  beliefs  were  measured  by  Teacher  Beliefs  Survey 
(McCombs  & Whisler,  1997).  This  original  survey  contained  35  four-point  rating  scale  items  (from  1-strongly 
disagree  to  4-strongly  agree).  The  factor  analysis  yielded  three  factors  and  29  items  (6  items  from  the  original 
survey  were  dropped).  The  three  factors  were  consistent  with  the  factors  defined  by  the  authors:  1)  learner- 
centered  beliefs  about  learners,  learning,  and  teaching  (LB),  such  as  the  item  “Students  have  more  respect  for 
teachers  they  see  and  can  relate  to  as  real  people,  not  just  as  teachers”,  2)  non-learner-centered  beliefs  about 
learners  (NLB-L),  such  as  the  item  “There  are  some  students  whose  personal  lives  are  so  dysfunctional  that  they 
simply  do  not  have  the  capability  to  learn”,  and  3)  non-learner-centered  beliefs  about  learning  and  teaching 
(NLB-TL),  such  as  the  item  “I  can’t  allow  myself  to  make  mistakes  with  my  students”.  The  reliability 
coefficient  alpha  for  the  three  factors  were  0.71,  0.70  and  0.71  respectively. 

Preservice  teachers’  technology  attitudes  (TA)  were  measured  by  computer  technology  attitude  survey 
that  was  developed  by  Francis  -Pelton  and  Pelton  (1996).  There  were  originally  42  five-point  Likert  scale  items 
(from  1-strongly  disagree  to  5-strongly  agree),  such  as  “Students  who  use  computer  will  have  difficulty  learning 
basic  skills”.  The  factor  analysis  identified  five  factors.  A total  of  38  items  were  retained.  The  reliability  alpha 
of  the  38  items  was  0.94.  The  alpha  for  the  five  factors  was  0.93,  0.87,  0.87,  0.72  and  0.76  respectively. 

Teacher  educators’  technology  uses  in  instruction  were  measured  in  two  parts.  One  part  was  the  frequency  of 
using  a variety  of  computer  tools  and  application  (software),  such  as  “Word  Processing”  and  “Database”.  The 
other  part  was  about  how  the  instructors  had  students  use  computer  technology.  This  part  contained  12  items 
that  fell  into  two  subscales.  One  subscale  included  8 items  that  reflected  using  technology  in  constructivist  way. 
They  were  adapted  from  the  objectives  for  computer  use  by  teachers  who  had  constructivist  teaching 
philosophy  (Becker,  1998)  and  from  constmctivist  instructional  goals  (Niederhauser  and  Stoddart,  2001).  The 
items  were  such  as  “expressing  themselves  in  writing”  and  “learning  to  work  collaboratively”.  The  other 
subscale  contained  4 items  that  reflected  using  technology  in  traditional  way  of  learning.  They  were  adapted 
from  the  objectives  for  computer  use  by  those  who  had  traditional  transmission  teaching  philosophy  (Becker, 
2000).  The  items  were  such  as  “mastering  skills  just  taught”  and  “learning  to  work  independently”.  Specifically, 
the  part  of  how  computer  technology  used  measured  the  frequency  that  the  instructors  have  students  use 
computer  technology  either  in  constructivist  way  (CW)  or  in  traditional  way  (TW).  All  the  items  were  in  the 
form  of  rating  scale  (from  1 - None  to  4 - High). 

Data  Collection 

At  the  beginning  of  the  spring  semester,  preservice  teachers’  pre -survey  was  administered  to  student 
participants  to  pretest  their  pedagogical  beliefs  and  attitudes  toward  technology.  At  the  end  of  the  semester,  a 
post-survey  that  was  similar  to  the  pre -survey  was  administered. 

At  the  end  of  the  semester,  a survey  was  administered  to  instructor  participants  to  examine  their  pedagogical 
beliefs  and  technology  uses  in  instruction.  All  the  surveys  were  put  online.  The  participants  were  informed  of 
the  web  address  of  the  surveys. 

Data  Analysis 

In  examining  the  relationship  between  teacher  educators’  beliefs  and  their  technology  uses,  the  Pearson 
product-moment  correlation  coefficient  r was  measured  between  each  of  the  three  beliefs  scores  (LB,  LB-L, 
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LB-TL)  and  the  score  for  each  subscale  of  computer  technology  uses  (software,  CW,  TW).  Positive  rvalue 
shows  positive  relationship  between  two  variables,  while  negative  r value  shows  negative  relationship  between 
two  variables.  P value  was  reported.  The  significance  level  was  set  at  .05. 

When  examining  the  relationship  between  teacher  educators’  beliefs  and  preservice  teachers’  beliefs, 
data  collected  from  100  student  participants  and  the  instructors  (N  = 17)  in  course  A,  B and  C was  analyzed. 
Sequential  multiple  regression  analysis  was  conducted  to  examine  each  beliefs  score  separately.  To  control  the 
influence  of  student  participants’  pre-existed  beliefs  prior  to  coming  to  teacher  education  program,  their  beliefs 
score  in  the  pre -survey  was  used  as  covariate  in  the  regression  analysis.  The  students’  post-survey  beliefs  score 
was  regressed  as  functions  of  instructors’  beliefs  score  with  the  students’  pre -survey  beliefs  scores  as  covariate. 
Since  not  all  the  student  participants  took  course  C,  dummy  variable  was  used  to  indicate  course  C instructor, 
with  “1”  indicating  having  this  instructor  and  “0”  indicating  not  having  this  instructor.  Specifically,  there  will 
be  three  models: 


Learner-centered  beliefs  (LB)  model 

y = Po  + P iXj  + P 2%2  + P2X3  + P 2X-4 

Non-learner-centered  beliefs  about  learners  (NLB-L)  model 

y = Po  + P A + p 2X2  + P 2x3  + p 2X4 

Non-learner-centered  beliefs  about  teaching  and  learning  (NLB- 
TL)  model 

y = Po  + P 1X1  + P 2X2  + P 2X3  + P 2X4 

In  each  model,  Xt  represents  students’  beliefs  score  in  pre -survey,  X2  represents  course  A instructors’  beliefs 
score,  ^represents  course  B instructors’  beliefs  score,  70  represents  the  indicator  of  course  C instructor’s 
beliefs,  and  y represents  students’  beliefs  score  in  post-survey. 


In  the  examination  of  the  relationship  between  teacher  educators’  uses  of  technology  and  preservice 
teachers’  attitudes  toward  technology,  similar  method  used  to  answer  the  second  research  question  was  applied. 
Sequential  multiple  regression  analysis  was  again  used.  Students’  technology  attitude  score  in  pre -survey  was 
used  as  covariate.  Two  models  were  identified: 


Models  Variables  included 


1.  Students’  technology  attitudes 
(TA)  are  predicted  by  the  frequency 
that  instructors  use  software 
(software)  and  the  frequency  of  their 
having  students’  use  technology  in 
constructivist  way  (CW) 

y = Po  + P/X,  + P2X?  + P2X?  + P 2X4+  P2X5  + p 2X(, 

Xj-  Students’  technology  attitude  score  in  pre-survey  (TA) 

X2-  software -Frequency  of  course  A instructors 

X3-  CW -Frequency  of  course  A instructors 

X4-  software -Frequency  of  course  B instructors 

X5-  CW -Frequency  of  course  B instructors 

X6  - Indicator  of  course  C instructor 

2.  Students’  technology  attitudes 
(TW)  are  predicted  by  the  frequency 
that  instructors  use  software 
(software)  and  the  frequency  of  their 
having  students’  use  technology  in 
traditional  way  (TW) 

y = Po  + P/X,  + P2X2  + P2X?  + P2X4+  P2X5  + P2X6 

X/-  Students’  technology  attitude  score  in  pre-survey  (TA) 

X2-  software -Frequency  of  course  A instructors 

X3-  TW -Frequency  of  course  A instructors 

X4-  software -Frequency  of  course  B instructors 

X5-  TW-Frequency  of  course  B instructors 

X5  - Indicator  of  course  C instructor 

Results 

Relationship  between  Teacher  Educators’  Beliefs  and  Technology  Use 

An  ANOVA  examination  found  that  graduate  instructors  and  faculty  members  were  different  in 
learner-centered  beliefs  (LB),  relationship  between  instructors’  beliefs  and  their  technology  uses  was  examined 
with  the  two  groups  of  instructors  separately. 

For  graduate  instructors  ( N = 18),  their  learner-centered  beliefs  (LB)  were  positively  related  to  their 
software  use  (r  = .47 ,p  = .05),  the  frequency  that  they  had  students  use  technology  in  both  constructive  way 
(CW)  (/•  = .57,  p = .01)  and  traditional  way  (TW)  (r  = .54,  p = .02).  Their  non-learner-centered  beliefs  about 
learners  (NLB-L)  were  negatively  related  to  their  software  use  (r  = -.51,  p = .03). 

Since  there  were  only  6 faculty  member  participants,  only  one  significant  result  was  found.  The 
frequency  that  these  faculty  members  had  students  use  technology  in  traditional  way  (TW)  was  positively 
related  to  their  non-learner-centered  beliefs  about  learners  (NLB-L)  (r  = .89,  p = .02). 

Relationship  between  Teacher  Educators’  Beliefs  and  Preservice  Teachers’  Beliefs 

When  examining  the  three  beliefs  models,  the  regression  analysis  of  the  learner-centered  beliefs  (LB) 
produced  a model  of  two  variables  that  best  predicted  students’  learner-centered  beliefs:  students’  pre-survey 
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beliefs  and  course  A instructors’  learner-centered  beliefs  (LB);R2  = .13,  F (2,  97)  = 7.15,/?  = .0013.  This  model 
accounted  for  13%  of  variance  in  students’  learner-centered  beliefs.  Since  students’  pre -survey  beliefs  score 
was  used  as  covariate,  the  prediction  of  course  A instructors’  learner-centered  beliefs  was  the  focus  of 
examination.  That  semi-partial  correlation  was  0.04  indicated  that  course  A instructors’  beliefs  can  help  to 
explain  4%  variance  in  the  students’  learner-centered  beliefs.  A summary  of  the  model  is  presented  as  the 
follows: 


Steps 

R1 

df 

dh 

P 

3 

sr 

1.  Students’  LB  in  pre -survey 

1 

.001* 

0.1 

2.  Course  A instructors’  LB 

1 

.048* 

0.04 

Note.  dj=  degree  of  freedom;  (3=standardized  regression  coefficient;  .y^semi  -partial  correlation 
*p<.05 


Relationship  between  Teacher  Educators’  Technology  Uses  and  Preservice  Teachers’  Technology 
Attitude 

When  regressing  the  students’  technology  attitudes  score  as  functions  of  the  instructors’  using  of 
software  and  having  students  use  technology  in  constructivist  way  (CW),  the  regression  analysis  produced  a 
model  of  two  variables  that  best  predicted  students’  technology  attitude:  students’  pre -survey  attitudes  score  and 
course  A instructors’  having  students’  use  technology  in  constructivist  way  (CW);  R2  = .62,  F (2,  97)  = 80.52,  p 
< .0001.  This  model  accounted  for  62%  variance  in  students’  technology  attitudes. 

When  regressing  the  students’  technology  attitudes  score  as  functions  of  the  instructors’  using  of 
software  and  having  students  use  technology  in  traditional  way  (TW),  two  variables  were  significant  predictors 
of  students’  technology  attitudes:  students’  pre -survey  score  and  course  A instructors’  having  students  use 
technology  in  traditional  way  (TW);  R2  = .62,  F (2,  97)  = 79.64,  p < .0001.  This  model  also  accounted  for  62% 
variance  in  students’  technology  attitudes. 

Since  students’  pre -survey  attitudes  score  was  used  as  covariate,  the  prediction  of  the  course  A 
instructors’  having  students  use  technology  in  constructivist  way  and  traditional  way  was  the  focus  of  interests. 
The  semi  -partial  correlation  was  0.02  for  the  two  significant  variables,  which  indicated  that  the  frequency  of  the 
instructors’  having  students  use  technology  in  either  constructivist  way  or  traditional  way  accounted  for  2% 
variance  respectively  in  students’  technology  attitudes.  The  following  tables  presented  the  summary  of  the 

regression  analysis. 

Model  1.  Students’  technology  attitudes  were  predicted  by  the  frequency  that  instructors  use  software  and 

their  having  students’  use  technology  in  constructivist  way  (CW) 

Variables  R 2 ~dj\  ~p  (3  ~r 

1.  Students’ technology  attitude  in  pre -survey  0.60  1 98  .00*  0.78  0.61 

2.  Course  A instructors’  CW -Frequency 0.62  1 97 .03*  0.14  0.02 

Model  2.  Students’  technology  attitudes  are  predicted  by  the  frequency  that  instructors  use  software  and 

their  having  students’  use  technology  in  traditional  way  (TW) 

Variables  7?  ~dfi  "dfc  ~P  (3  ~r 

1.  Students’ technology  attitude  in  pre -survey  0.60  1 98  .00*  0.78  0.61 

2.  Course  A instructors’  TW -Frequency 0.62  1 97 .04*  0.13  0.02 

Note.  d/=degree  of  freedom;  (3=standardized  regression  coefficient;  s^semi  -partial  correlation 
*p<.05 


Discussion  and  Implication 

The  findings  of  this  study  revealed  that  graduate  instructors  who  had  more  learner-centered  beliefs 
tended  to  use  various  software  programs  more  frequently  and  have  students  use  technology  more  frequently  in 
constructivist  ways.  The  graduate  instructors  who  had  more  non-learner-centered  beliefs  about  learners  tended 
to  use  software  programs  less  frequently.  It  is  interesting  to  note  that  those  who  had  more  learner-centered 
beliefs  also  tended  to  have  students  use  technology  more  frequently  in  traditional  ways.  This  indicated  that 
teacher  educators’  use  of  technology  was  not  in  a simple  dimension.  For  some  reason,  those  who  have  more 
learner-centered  beliefs  would  have  students  use  technology  not  only  in  constructivist  way  but  also  in 
traditional  way.  The  inconsistency  between  teachers’  beliefs  and  their  use  of  technology  was  reported  in  Ertmer, 
Gopalakrishnan  and  Ross’s  study  (2001).  In  this  study,  the  researchers  conducted  an  exploratory  study  with 
seventeen  school  teachers  who  considered  themselves  to  be  exemplary  technology  users  to  examine  their 
pedagogical  beliefs  and  classroom  practices.  It  was  found  that  although  most  of  the  teachers  reported  to  have 
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constructivist  pedagogical  philosophy,  only  five  of  them  implemented  in  terms  of  best  practice  identified  by  the 
literature.  They  articulated  constructivist  views  on  technology  integration,  however,  they  used  both 
constructivist  and  traditional  methods  to  implement  their  teaching  practice.  The  reason  that  there  is 
inconsistency  between  teachers’  beliefs  and  practice  is  the  complexity  in  the  classroom  situation.  The 
contextual  factors  in  the  classroom  teaching  may  prevent  teachers  from  transforming  their  beliefs  into  practice 
and  implementing  what  they  would  do  theoretically  (Fang,  1996). 

Graduate  instructors  shared  the  responsibilities  of  educating  preservice  teachers  in  teacher  education 
program  and  could  be  major  resources  of  candidates  of  faculty  members  in  teacher  education  field.  As  such,  the 
examination  of  this  group  of  participants  was  very  important  in  the  exploration  of  teacher  educators’  beliefs  and 
technology  use. 

Due  to  the  small  number  of  faculty  member  participants,  only  one  significant  result  was  found  in 
faculty  member  participants’  data.  Those  who  had  more  non-learner-centered  beliefs  about  learners  tended  to 
have  students  use  technology  more  frequently  in  traditional  way.  To  further  explore  the  relationship  between 
teacher  educators’  beliefs  and  technology  uses,  in  future  study,  it  is  necessary  to  include  more  instructor 
participants,  especially  faculty  member  participants. 

In  the  examination  of  the  relationship  between  teacher  educators’  beliefs  and  preservice  teachers’ 
beliefs,  this  study  revealed  that  teacher  educators’  learner-centered  beliefs  could  be  able  to  influence  preservice 
teachers’  learner-centered  beliefs  over  a semester,  which  was  found  between  student  participants  and  the 
instructors  in  course  A.  Richardson  (1996)  commented  that  when  preservice  teachers  first  came  into  teacher 
education  program,  they  already  had  certain  form  of  beliefs  based  on  their  own  previous  experience  as  students. 
These  beliefs  were  deep-seated,  therefore,  it  was  hard  to  have  their  beliefs  be  impacted.  This  is  true  in  the  aspect 
that  in  current  study,  students’  pre -survey  beliefs  scores  were  always  the  predictors  of  their  post-  survey  beliefs 
scores.  Since  their  pre -survey  was  conducted  at  the  very  beginning  of  the  semester,  the  pre  - survey  scores  can 
reflect  their  beliefs  prior  to  their  coming  to  the  teacher  education  program.  In  addition,  the  students’  non- 
learner-centered  beliefs  were  not  found  to  be  influenced  by  the  instructors’  beliefs.  However,  if  previous 
instruction  that  students  received  could  help  in  the  development  of  their  beliefs,  there  is  no  reason  to  deny  that 
the  instruction  that  preservice  teachers  receive  in  teacher  education  program  could  have  influence  on  their 
beliefs  about  teaching  and  learning.  In  addition,  their  study  in  teacher  education  program  and  their  status  of 
being  future  teachers  could  make  them  think  about  teaching  and  learning  more  seriously  and  systematically.  In 
this  study,  In  comparison  with  the  fact  that  students’  pre  - learner-centered  beliefs  accounted  for  10%  variance  in 
their  post-  learner-centered  beliefs  (sr  = .10),  the  4%  (sr  = .04)  variance  accounted  by  course  A instructors’ 
learner-centered  beliefs  in  one  semester  did  indicate  that  teacher  educators’  learner-centered  beliefs  can  have 
influence  on  preservice  teachers’  learner-centered  beliefs. 

This  study  revealed  that  the  student  participants’  attitudes  toward  technology  could  be  predicted  by  the 
frequencies  of  course  A instructors’  having  students  use  technology  in  constructivist  way  and  traditional  way. 
Although  these  two  aspects  of  technology  use  accounted  for  just  2%  variance  respectively  in  students’ 
technology  attitude  (sr  = .02,  sr  = .02),  they  were  significant  and  did  help  to  explain  the  students’  attitude  score 
in  post-survey.  The  examination  of  the  descriptive  data  of  the  instructors’  technology  use  indicated  that  the 
frequency  of  their  having  students’  use  technology  for  either  constructivist  way  or  traditional  way  of  obtaining 
knowledge  was  barely  moderately  (several  times  a semester).  Given  this , the  2%  variance  cannot  be  discounted. 

Aiken  (1980)  pointed  out  that  many  attitudes  were  “learned  by  observing  the  activities  of  people  who 
are  perceived  as  significant’’  (p.  16).  Thinking  about  the  fact  that  course  A instructors  had  influence  on  the 
students’  learner-centered  beliefs,  one  can  say  that  course  A instructors’  instruction  impressed  the  students  more 
than  the  other  instructors’  instruction.  Therefore,  it  is  reasonable  to  think  that  the  development  of  students’ 
attitudes  toward  technology  was  more  the  result  of  observing  how  course  A instructors  used  technology.  In 
other  words,  course  A instructors’  use  of  technology  can  predict  students’  attitudes  toward  technology. 

According  to  Wetzel  (2002),  “For  instructional  technology  to  be  successfully  implemented,  teacher 
beliefs  and  values  need  to  shift.  If  they  do  not,  the  desired  implementation  and  integration  of  instructional 
technology  in  education  will  not  occur  on  a broad  scale’’  (p.  46).  To  better  facilitate  professional  development 
for  teacher  educators  and  better  prepare  tomorrow’s  teachers  to  integrate  technology  effectively  in  classrooms, 
it  is  necessary  to  examine  teacher  educators’  beliefs  and  their  uses  of  technology,  and  how  the  two  variables  are 
related  to  preservice  teachers’  beliefs  and  attitudes  toward  technology  use.  As  an  exploratory  effort,  this  study 
helped  to  enrich  our  knowledge  about  helping  teacher  educators  to  use  technology  in  teacher  preparation 
courses  and  engage  in  preparing  technology -using  prospective  teachers. 

Due  to  the  limit  of  time  and  resources,  this  study  was  conducted  in  one  semester  period.  In  the  future 
when  time  and  funding  permitting,  further  study  can  be  conducted  to  explore  how  preservice  teachers’ 
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pedagogical  beliefs  and  their  attitudes  toward  technology  develop  over  the  whole  period  in  teacher  education 
program.  This  may  help  the  researchers  and  practitioners  to  learn  more  about  the  growth  of  preservice  teachers 
and  better  prepare  them  for  their  future  teaching. 
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Introduction 

Articulating  a clear  definition  and  process  of  design  research  is  a current  and  prominent  topic  among 
educational  researchers.  Design  studies  involve  complex  interactions  and  feedback  cycles  that  can  significantly 
blur  the  roles  of  researchers,  teachers,  curriculum  developers,  instructional  designers  and  assessment  experts 
(Kelly  and  Lesh,  2000).  As  educational  researchers  struggle  to  clarify  this  research  method,  they  raise 
significant  questions  such  as  how  is  design  research  different  from  the  process  of  design?  What  are  appropriate 
methods  and  processes  that  can  be  used  in  design  research?  How  do  we  systematically  create,  test  and 
disseminate  design  or  teaching  interventions  that  will  have  maximum  impact  on  practice? 

Kelly  and  Lesh  (2000)  and  others  (Cobb,  Confrey,  diSessa,  Lehrer,  & Schauble,  2003;  Collins,  1999; 
The  Design-Based  Research  Collective,  2003)  advocate  that  these  emerging  methods  call  for  the  articulation  of 
new  processes  and  criteria  including  factors  such  as  the  usefulness  and  usability  of  knowledge,  its  shareability, 
and  marketability,  how  well  it  disseminates  and  the  extent  to  which  it  positively  impacts  practice.  Sabelli 
(personal  communication.  May  16,  2002)  cites  a need  for  organizational  structure  and  protocol  for  the  diffusion 
of  research  into  practice  and  states  that  educational  research  situations  are  extremely  complex  systems  that  can 
benefit  from  integrated  system  research  strategies. 

There  is  a need  for  comprehensive  models  to  guide  design  research  addressing  the  process  of 
designing,  developing  and  assessing  the  impact  of  an  educational  innovation.  Kelly  (2004)  points  to  the  need  for 
enhanced  guidance  and  specification  in  challenging  design  researchers  to  “clarify  in  which  stage  and  for  what 
purposes  methods  are  appropriate  and  inappropriate’'  to  begin  to  “explore  the  necessary  aspects  of  frameworks 
and  findings  of  design  studies”  (p.  125).  Researchers  and  practioners  in  the  field  of  instructional  design  and 
complementary  design  domains  such  as  product  design,  usage-centered  design  and  innovation  development 
have  much  to  offer  in  the  emergent  definition  and  process  of  design  research.  We  present  a synthesis  of  existing 
design  and  research  processes  offering  a guiding  framework  that  goes  beyond  the  individual  domains 
represented  and  challenges  researchers  to  provide  improved  articulation  of  design  research  processes  and 
consider  the  entire  scope  of  research  from  initial  conceptualization  to  diffusion  and  adoption. 

The  integrative  learning  design  study  (ILD)  framework  initally  presented  in  previous  article  (Bannan- 
Ritland,  2003)  and  elaborated  in  this  paper  comprises  a complementary  process  model  based  on  several  years  of 
implementation  in  an  instructional  design  evaluative  context.  Building  on  processes  from  multiple  fields,  ILD 
present  a “meta-methodological”  view  that  attempts  to  integrate  the  best  of  design,  research  and  diffusion  of 
educational  innovations  across  a program  of  research.  Traditional,  behaviorist  views  of  computer-assisted 
instruction  development  while  offering  valuable  iterative  processes,  do  not  fully  address  the  complexity 
inherent  in  a cognitively  informed,  contextually -based  and  data-driven  decision-making  process  situated  in 
educational  practice.  In  response,  the  ILD  model  attempts  to  provide  a comprehensive  yet  flexible  guiding 
framework  that  positions  design  research  as  a socially -constructed,  contextualized  process  of  producing  the 
most  educationally  effective  product  that  has  the  best  chance  to  be  used  in  the  classroom  The  framework 
attempts  to  move  past  isolated,  individual  efforts  of  design  research  by  clearly  articulating  a logically -ordered 
structural  frame  that  considers  the  full  spectrum  of  applied  and  experimental  research  methodology  in 
advancing  toward  systemic  impact  in  education  and  may  be  applied  in  a variety  of  contexts.  Collins  (1990; 
1993)  advocates  for  a similar  overt,  systematic  methodology  for  conducting  design  experiments.  He  refers  to 
many  design  efforts  that  do  not  adhere  to  systematic  and  rigorous  processes  stating: 

“When  designing  a learning  environment,  whether  computer  based  or  not,  there  are  a multitude  of 
design  decisions  that  must  be  made.  Many  of  these  design  decisions  are  made  unconsciously  without  any 
articulated  view  of  the  issues  being  addressed  or  the  tradeoffs  involved.  It  would  be  better  if  these  design 
decisions  were  consciously  considered,  rather  than  unconsciously  made  (1993,  p.l).” 

Even  in  the  best  applications  of  systematic  instructional  design,  many  decisions  are  made  in  the 
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absence  of  a clear  rationale  and  analysis  of  data.  In  attempting  to  combine  research  and  design  processes,  design 
research  strives  to  build  knowledge  about  teaching  and  learning  while  also  engineering  an  effective  learning 
environment.  In  this  manner,  design  research  is  a unique  form  of  educational  research  because  it  requires 
investigators  to  pursue  several  questions  across  a program  of  research,  often  in  concert.  Barab  and  Squire 
(2004)  describe  this  multi-level  effort  of  design  research  as  attempting  to  promote  pragmatic  change  in  local 
educational  contexts  while  also  advancing  theoretical  understanding  about  learning  and  cognition.  Fishman, 
Marx,  Blumenfeld,  Krajcik,  and  Soloway  (2004)  propose  a series  of  demanding  research  questions  for  the  field 
related  to  the  sustainability  and  diffusion  of  design  research  efforts  that  consider  complex  factors  such  as 
teacher  learning,  assessment,  technology  planning  and  organizational  structure.  Given  the  complexity  of  these 
multi-layered  efforts,  it  is  not  surprising  that  design  research  demands  multiple  research  questions  and  methods 
(Middleton,  Gorard,  Taylor  & Bannan-Ritland,  2004).  However,  to  promote  “...a  theory -driven  process  of 
designing  and  a data-driven  process  of  refining”  (National  Research  Council,  2002,  p.122)  educational  artifacts 
that  generate  knowledge  as  well  as  address  pragmatic  problems  in  education,  better  articulation  of  the  types  of 
questions  and  methods  appropriate  for  specific  points  in  the  process  are  needed. 

The  ILD  process  presents  one  step  toward  a systematic  framework  for  the  articulation  and 
documentation  of  common  phases  and  complementary  stages  based  on  multiple  design  and  research  processes 
promoting  more  conscious  design  research  (Collins,  1990;  1999).  It  is  presented  here  as  a starting  point  for 
researchers  to  consider  with  the  goal  of  eliciting  questions,  suggestions,  limitations  and  criteria  that  may  need  to 
be  considered  as  researchers  struggle  with  the  implications  of  this  emerging  form  of  educational  research.  In 
this  paper,  we  briefly  describe  the  progression  of  a design-based  research  study  currently  underway  that 
illustrates  an  ILD.  Then,  the  general  ILD  framework  is  abstracted  from  this  example  primarily  represented 
through  Figure  1 (Bannan-Ritland,  2003)  with  related  implications  and  conclusions.  The  ILD  example  is 
described  according  to  broad  phases  including  1)  the  informed  exploration  phase;  2)  the  enactment  phase;  3)  the 
evaluation  phase;  and  4)  the  dissemination  phase  that  emerged  from  a recent  design  study,  Literacy  Access 
Online. 


LiteracyAccess  Online  - An  Integrative  Learning  Design  Study 

The  LiteracyAccess  Online  (LAO)  project1  provides  an  example  of  the  application  of  the  integrative 
learning  design  framework  illustrating  the  intersection  and  systematic  expressbn  of  multiple  design  and 
research  methods.  LiteracyAccess  Online  is  an  effort  to  utilize  Web-based  technology  to  provide  support  for 
teachers,  tutors,  and  parents  (literacy  facilitators)  in  addressing  literacy  goals  for  all  children  with  a particular 
focus  on  those  with  disabilities.  LAO  provides  a technology -based  learning  environment  that  promotes  the  use 
of  specific  literacy  strategies  for  the  improvement  of  tutoring  and  reading  performance  as  the  child  and 
facilitator  collaboratively  engage  in  the  process  of  reading  online. 

Related  to  the  instructional  and  performance  support  goals  of  LAO,  it  was  well  established  in  the 
literature  that  one-to-one  tutoring  is  one  of  the  most  effective  forms  of  instruction  for  improving  reading 
achievement  but  increased  success  often  depends  upon  the  skill  of  the  tutor  or  facilitator  and  the  establishment 
of  consistent  roles  and  expectations  (Wasik,  1998).  What  was  not  well  known  was  whether  a technology 
system  could  support  the  complex  interaction  between  parents,  teachers,  or  tutors  and  children  by  providing  a 
consistent  environment  and  reading  support  for  both  members  of  the  dyad. 

The  Informed  Exploration  Phase 

The  research  objective  of  the  LAO  integrative  learning  design  study  was  to  investigate  the  nature  of 
interaction,  tool  use  and  activities  that  occur  as  literacy  facilitators  and  children  simultaneously  engage  in  the 
reading  process  using  embedded  metacognitive  strategy  and  assistive  technology  supports.  This  objective 
resulted  from  extended  investigation  into  the  provision  of  literacy  support  for  facilitators  and  children.  Initial 
explorations  into  target  audience  and  stakeholder  perceptions,  related  products  and  literature  and  documentation 
of  the  complex  nature  of  supporting  literacy  revealed  many  plausible  paths  for  design  research. 

The  interdisciplinary  research  team  involved  in  the  LAO  project  were  charged  with  determining  the 
research  direction  and  consisted  of  educational  researchers,  teachers,  graduate  students,  content  experts  in 
literacy,  special  education  and  assistive  technology  as  well  as  parents  involved  in  an  advocacy  group  for 
children  with  disabilities.  The  research  focus  evolved  from  the  team’s  perceived  lack  of  support  for  children 
who  were  struggling  with  the  literacy  process  based  on  direct  observations  of  this  problem  in  both  classroom 
and  home  environments.  Aligned  with  Confrey  and  Lachance’s  (2000)  notion  of  drawing  key  inferences  from 
dissatisfaction  with  current  educational  practices  and  direct  experiences  with  children,  initial  theoretical 
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conjectures  were  developed  that  advocated  for  reading,  writing  and  assistive  technology  support  for  children 
with  or  without  disabilities  to  increase  their  engagement  and  performance  in  literacy. 

While  these  initial  theoretical  conjectures  provided  a central  premise  and  broad  direction  for  design 
research,  more  information  was  needed  to  refine  these  conjectures  resulting  in  a comprehensive  needs  analysis 
and  literature  review  that  provided  a firm  foundation  for  the  intended  design.  Extensive  exploration  into 
appropriate  literacy  strategies,  tutorial  programs  and  processes,  surveys  of  experts,  teachers  and  parents  as  well 
as  observations  of  children  and  facilitators  engaged  in  a literacy  experience  all  informed  this  phase  of  the 
research.  Many  potential  design  research  directions  were  considered  based  on  the  initial  conjectures,  however, 
data  drawn  from  conducted  interviews,  direct  experience  with  potential  research  participants  and  literature 
review  converged  and  pointed  the  team  in  a particular  direction. 

A prominent  theme  that  emerged  across  initial  interviews,  surveys  and  observations  with  experts, 
parents,  teachers  and  children  revealed  that  literacy  facilitators  had  a crucial  role  in  providing  support  for 
children  struggling  to  gain  literacy  skills  and  the  question  remained  how  to  best  support  this  role.  These 
findings  and  related  literature  provided  insight  for  informed  theory  and  improved  conjectures  based  on  results 
indicating  that  1)  children  can  but  often  do  not  use  effective  metacognitive  reading  strategies;  2)  explicitly 
teaching  these  strategies  can  greatly  enhance  children’s  comprehension  of  text;  3)  teachers  (as  well  as  other 
literacy  facilitators)  need  to  be  trained  in  how  to  provide  cognitive  structure  for  their  students  so  that  children 
can  learn  to  guide  their  own  generative  processes  in  reading;  and  4)  one-to-one  tutoring  is  one  of  the  most 
effective  forms  of  instruction  for  improving  reading  achievement  but  increased  success  often  depends  upon  the 
skill  of  the  tutor  or  facilitator  and  the  establishment  of  consistent  roles  and  expectations  (Wittrock,  1998; 
National  Reading  Panel,  2000;  Wasik,  1998). 

This  exploration  into  the  literature  and  perspectives  of  those  involved  in  these  issues  greatly  refined 
our  initial  conjectures  and  dramatically  changed  our  intended  design  direction  for  this  research  from  a didactic, 
tutorial,  child-focused  intervention  to  a collaborative,  story -based  reading  experience  providing  embedded 
metacognitive  strategy  support  for  both  the  literacy  facilitator  and  the  child’s  use.  The  rationale  for  this 
research  direction  was  documented  in  a comprehensive  needs  analysis  that  detailed  the  data  collection, 
conclusions  and  related  literature  review  (see  Literacy  Access  Online  Phase  Two,  2000). 

The  next  stage  of  our  research  involved  the  analysis  and  description  of  the  range  of  learners  and 
facilitators  that  would  potentially  use  the  LAO  system.  Direct  experience  with  4th-8lh  grade  children  with  or 
without  disabilities,  teachers,  tutors,  and  parents  provided  data  that  characterized  our  audience.  These 
descriptions  were  depicted  as  role  models  (Constantine  & Lockwood,  1999)  or  personas  (Cooper,  1999)  that 
comprised  abstract  composite  profiles  of  audience  characteristics  gleaned  from  actual  interviews  and 
observations  and  provided  a focal  point  for  design.  Role  models  or  personas  are  similar  to  Graue  and  Walsh’s 
(1998)  qualitative  vignettes  that  strive  to  capture  the  substance  of  a setting,  person  or  event  to  communicate  a 
central  theme  of  qualitative  data,  based  on  mu  ltiple  direct  observations. 

Exploring  the  nature  of  the  identified  educational  problem,  related  products  and  literature  as  well  as 
creating  and  refining  theoretical  conjectures  and  descriptions  of  the  audience  provided  an  informed  perspective 
for  grounded  design  of  a learning  environment  based  on  articulated  theory.  These  activities  resulted  in  specific 
research  artifacts  including  a needs  analysis  that  contained  an  extensive  literature  review,  an  articulated  and 
congruent  design  and  research  direction  and  detailed  audience  analysis  based  on  qualitative  and  quantitative 
data.  These  documents  were  housed  on  a project  Web  site  (see  for  example,  LAO  Phase  Four,  2000;  LAO 
Phase  Six,  2003)  that  provided  a communication  mechanism  between  team  members  as  well  as  an  archive  of 
shareable  design  research  processes,  products  and  evidentiary  data. 

The  Enactment  Phase 

The  embodiment  of  the  results  of  our  informed  exploration  and  theories  about  providing  literacy 
support  for  children  in  a usable  learning  environment  were  collaboratively  constructed  across  several  stages  and 
feedback  cycles  culminating  in  a Web-based  prototype.  The  initial  design  of  the  LAO  learning  environment 
resulted  directly  from  the  design  implications  articulated  in  the  previous  phase  of  exploration,  analyses  and 
review.  These  implications  were  translated  into  an  articulated  prototype  initially  developed  by  building  an 
abstract,  paper-based  model  of  the  system  for  researcher  and  teacher  input  according  to  procedures  adapted 
from  usage-centered  design  (Constantine  & Lockwood,  1999).  The  abstract  representation  of  the  LAO  learning 
environment  allowed  for  a fluid,  flexible  instantiation  of  theory  through  design  that  was  described  and  presented 
to  facilitators,  teachers,  an  expert  panel  and  researchers  for  feedback.  Abstract  modeling  of  the  instantiated  or 
enacted  design  provided  opportunities  for  input  and  co-construction  of  LAO  with  several  audience  members 
prior  to  the  more  time  -intensive  computer-based  production  of  the  learning  environment. 
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After  several  iterative  cycles  of  feedback  and  revision  of  the  articulated  prototype,  detailed  design  then 
took  place  through  the  production  of  flowcharts,  technical  specifications  and  storyboards  eliciting  feedback  at 
each  stage  from  the  entire  team  and  leading  to  the  creation  of  a Web-based  prototype.  The  methods  of  data 
collection  employed  at  this  stage  included  designer  logs  posted  on  the  project  Web  site,  expert  panel  reviews  of 
the  design  and  documented  reviews  of  the  design  by  content  experts,  audience  members  and  the  research  team. 

The  Evaluation  Phase 

Once  a physical  prototype  was  in  place,  formative  testing  could  commence  and  characterized  the 
highly  iterative  nature  of  the  evaluation  phase  as  it  informed  and  refined  both  our  theories  and  redesign  efforts. 
The  complex  interactions  between  facilitators  and  children  that  can  occur  in  multiple  settings  formed  the  series 
of  studies  in  LAO  examining:  1)  parent-child  dyads  in  an  informal  setting  with  extensive  involvement  by 
researchers;  2)  parent-child  dyads  in  a structured  workshop  experience  supported  by  researchers  and;  3)  pre  - 
service-teacher  dyads  in  a field  trial  closely  modeling  authentic  conditions.  When  a fully  functioning  prototype 
was  not  yet  available,  studies  attempted  to  closely  mimic  the  tasks  that  would  be  embedded  in  LAO.  The  data 
gathering  across  these  three  studies  incorporated  observations,  interviews,  child  and  parent  journal  entries, 
videotaped  use  of  system  and  pre  - and  post-online  surveys.  This  multi-tiered,  multi- method  evaluation  scheme 
generated  useful  knowledge  and  subsequent  results  from  each  stage  of  inquiry  were  then  cycled  into  changes  in 
theoretical  conjectures,  research  design  as  well  as  system  design.  This  process  revealed  insights  into  the  core 
principles  that  may  support  the  collaborative  learning  and  implementation  of  metacognitive  processes  by 
literacy  facilitators  and  children.  Among  the  multiple  findings  of  these  initial  studies  were  that  1)  parent  literacy 
facilitators  could  develop  greater  awareness  and  skill  in  inplementing  reading  activities  and  identify  supports 
for  their  child  in  a structured  setting;  2)  children  showed  improvement  in  literacy  skills  using  technology-based 
support  when  participating  in  a guided  workshop  environment;  and  3)  pre  -service  teachers  felt  that  the 
strategies  and  activities  embedded  in  the  LAO  environment  facilitated  children’s  comprehension,  motivation 
and  interest  when  working  with  them  in  this  environment.  More  rigorous  evaluations  are  planned 
systematically  increasing  number  of  participants  and  varying  contexts.  These  studies  involve  detailed  tracking 
of  computer-based  activities  of  the  dyads  in  school  and  home  settings,  assessment  of  facilitator  and  child  use  of 
metacognitive  strategies  prior  to  using  LAO  and  pre  - and  post  comprehension  measures  after  several  weeks  of 
using  the  system. 

The  Dissemination  Phase 

The  last  phase  of  this  design-based  research  effort  involves  disseminating  LAO  into  the  broad 
educational  system.  Although  the  LAO  research  has  not  yet  fully  progressed  through  this  stage,  initial 
explorations  in  this  area  have  yielded  some  unique  insights  into  the  dissemination  process.  LAO,  as  a Web- 
based  learning  environment,  affords  the  opportunity  to  publish  current  working  prototypes  online  for  open  use 
and  input  that  has  resulted  in  an  early  and  unique  diffusion  and  adoption  process  begun  prior  to  the  completion 
of  a fully  functioning  system.  While  still  in  development,  we  have  tracked  over  100  potential  adopters  that  have 
discovered  and  explored  the  LAO  site.  The  profiling  and  data-base  capabilities  of  the  site  permit  tracking  and 
analysis  of  this  information  that  has  provided  detailed  information  on  potential  adopters  of  the  system  providing 
significant  insight  and  impact  on  sources  for  our  later  diffusion  efforts.  We  plan  to  incorporate  more 
sophisticated  computer-based  data  collection  and  analysis  techniques  such  as  datamining  (Tsantis  & Castellani, 
2001)  that  may  yield  even  more  insights  into  early  adopters’  behaviors,  profiles  and  use  of  this  new  tool.  We 
have  just  begun  to  publish  our  results  of  the  design  based  research  conducted  related  to  LAO  in  traditional 
academic  journals  and  non-traditional  Web  publishing  that  provide  avenues  for  additional  forms  of  review  and 
evaluation.  The  results  of  our  initial  studies  have  prompted  new  research  directions  such  as  exploring  the 
interaction  of  an  online  community  for  parents  of  children  with  disabilities  incorporated  in  the  LAO 
environment.  Given  the  iterative  nature  of  this  type  of  research,  it  is  highly  likely  that  determining  the 
consequences  of  the  LAO  design  research  effort  will  yield  new  theoretical  and  applied  questions  that  will 
prompt  the  entire  process  once  again. 

Characteristics  and  Implications  of  the  Integrative  Learning  Design  Studies 

Framework 

The  LAO  research  represents  the  combination  of  existing  empirical  methods  and  applied  practices  that 
characterize  the  ILD  framework  (see  Figure  1)  incorporating  the  creativity  of  design  communities  with 
integration  of  and  adherence  to  appropriate  quantitative  and  qualitative  methodology  in  education.  Drawing 
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heavily  upon  traditions  of  instructional  design  (Shrock,  1991),  product  design  (Urlich  & Eppinger,  2000), 
usage-centered  design  (Constantine  & Lockwood,  1999),  and  diffusion  of  innovations  (Rogers,  2003)  as  well  as 
established  educational  research  methodologies  (Isaac  & Michael,  1995),  the  ILD  approach  provides  a 
combination  of  processes  that  in  total  represent  an  emergent  perspective  for  viewing  the  spectrum  of  design 
research. 

Mapping  the  individual  stages  in  each  phase  presented  in  Figure  1 illustrates  the  limitations  of 
individual  design  or  research  processes.  For  example,  the  basic  educational  research  process  demonstrates 
pronounced  emphasis  on  the  initial  informed  exploration  phase  that  promotes  the  development  of  theory  or 
hypotheses  and  a lack  of  attention  to  the  enactment  phase  that  involves  development  of  educational 
interventions  characterizing  the  primary  difference  between  basic  research  and  design  research.  Also 
incomplete  are  the  design  processes  that  do  not  clearly  integrate  empirical  methods  of  reviewing  the  literature  or 
formulating  a research  design. 

The  Informed  Exploration  phase  of  the  LAO  design  research  reflected  in  the  ILD  framework  and  other 
design  research  cycles  or  models  resulted  in  a general  design  and  research  direction  investigating  technology- 
based  reading  strategy  support  in  collaborative  interaction  between  a literacy  facilitator  and  child.  While  the 
team’s  explicated  theoretical  conjecture  that  a well-designed  Web -based  system  may  promote  quality 
interactions  and  a high  level  of  reading  guidance  by  the  facilitators,  the  specifics  of  this  system  were  yet  to  be 
determined.  Initially  establishing  clear  learning  targets  or  objectives  for  the  enacted  design  allows  the  design 
researcher  to  characterize,  describe  and  eventually  test  the  intervention  based  on  more  specific  learning  criteria. 

However,  the  integration  of  processes  represented  at  the  top  of  Figure  1 show  that  the  series  of  actions 
reflective  of  different  domains  involved  in  design  and  research  are  highly  complementary  and  can  be 
synthesized  into  a comprehensive,  highly  iterative  design  research  cycle  that  was  followed  in  the  LAO  research. 
The  individual  processes  have  identified  limitations.  Limitations  will  certainly  also  exist  for  the  ILD 
framework,  however,  it  is  through  application  in  different  design  research  contexts  that  these  will  begin  to  be 
unveiled  and  begin  to  inform  the  process  presented  here. 

The  LAO  project  employed  evaluation  procedures  from  diffe  rent  fields  in  the  products’  life  cycle, 
progressing  from  high  levels  of  involvement  by  the  evaluators  to  authentic  contexts  and  conditions.  The 
questions  at  the  Evaluation  phase  for  the  LAO  project  were  how  to  ensure  that  the  design  was  usable,  effective 
and  that  the  enacted  theory  of  design  had  local  validity  or  relevance  to  the  potential  target  audience  and  context. 
Methods  were  employed  based  on  usability  testing  and  formative  evaluation  (Rubin,  1994;  Tessmer,  1993).  The 
LAO  project  has  gone  through  at  least  four  iterative  cycles  of  development,  usability  and  formative  evaluation 
testing.  During  each  evaluation  cycle,  different  research  methods  were  used  to  characterize  the  learning 
context,  assess  the  interaction  between  literacy  facilitator  and  child  and  determine  learning/performance 
outcomes  for  the  participants.  In  product  development  work,  the  testing  and  refinement  phase  is  characterized 
by  field  testing  the  product  and  implementing  any  necessary  design  or  assembly  process  changes.  Similarly, 
usability  testing  involves  identifying  weaknesses  in  the  product’s  interface  and  function  to  provide  a design  that 
fits  the  human  user’s  capabilities  and  natural  work  processes.  In  the  field  of  instructional  design  evaluation 
processes  includes  several  stages  of  formative  testing  different  from  usability  testing  in  that  formative 
evaluation  focuses  on  “. . .the  strengths  and  weaknesses  of  the  instruction”  rather  than  the  interface  ‘‘to  improve 
its  effectiveness  and  appeal”  (Tessmer,  1993,  p.  1 1).  The  formative  evaluation  process  includes  several  types 
including  expert  review,  one-to-one  evaluation,  small  group  evaluation  and  field  testing. 

The  impact  of  a systematic  process  of  conducting  design  research  that  incorporates  the  exploration  and 
articulation  of  design  direction  and  rationale,  documentation  of  data-driven  decision-making  and  multi-tiered, 
multi- method  evaluations  appropriate  for  specific  phases  of  the  research  and  design  process  remains  to  be  seen 
in  education.  The  LAO  example  briefly  presented  here  attempts  to  1)  characterize  the  informed  exploration  and 
evolution  of  significant  theoretical  conjectures  based  on  the  convergence  of  multiple  sources  of  data;  2) 
describe  a progressive,  socially -mediated  design  process  that  aligns  with  and  operationalizes  theory  and  is  fluid 
and  flexible;  as  well  as  3)  present  a systematic  and  graduated  scaling  up  of  situated  evaluation  in  multiple 
contexts.  The  example  also  provides  an  opportunity  for  capitalizing  on  small  amounts  of  initial  data  yielded 
from  highly  accessible  Web-based  learning  environments  in  a “proto-diffusion”  form  of  evaluation. 

Shavelson,  Towne,  Phillips  and  Feuer  (2003)  advocate  that  design  studies  are  process  focused  in  nature 
seeking  to  trace  participant’s  patterns  of  learning  as  well  as  the  inpact  of  the  educational  intervention  on  that 
learning.  We  would  argue  that  the  ILD  framework  subsumes  this  perspective  while  also  offering  a process 
focused  approach  for  researchers  to  trace  their  learning  through  the  patterns  of  data  generated  in  a design  study 
to  inform  theory  and  provide  documented  evidence  or  warrant  of  educational  impact  based  on  the  integration  of 
established  design  and  research  methodologies.  The  nature  of  the  educational  phenomenon  under  study  and 
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number  of  iterative  cycles  may  vary,  however,  the  ILD  articulates  a systematic  process  that  may  promote  more 
conscious,  data-driven,  decision-making  that  researchers  can  follow  and  document. 


Conclusion 

This  paper  has  presented  a brief  example  and  introduction  to  the  ILD  framework  that  comprises  a 
meta-methodological  view  of  the  design  research  process  in  an  attempt  to  elucidate  common  phases  and  stages 
in  this  specific  research  methodology.  The  framework  initially  presented  in  previous  work  (Bannan-Ritland, 
2003)  and  elaborated  here  is  presented  to  begin  to  establish  common  terminology  and  processes  that  can 
promote  conscious  design  research.  Most  importantly,  the  ILD  framework  is  an  attempt  to  provide  a roadmap 
for  future  design  researchers  to  investigate,  articulate,  document  and  inform  educational  practice. 

* I am  greatly  indebted  to  Dr.  Anthony  E.  Kelly  whose  insights  and  feedback  on  this  article  were  invaluable  in 
extending  my  thinking  in  this  area. 

1 The  Literacy  Access  Online  (LOA)  project  is  supported  by  the  Office  of  Special  Education  Programs  in  the 
Department  of  Education  Steppingstones  of  Technology  Innovation  for  Students  with  Disabilities  Grant 
CFDA84.327A. 
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Figure  1:  Integrative  Learning  Design  study  (ILD)  framework. 
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Sustaining  Motivation  for  Self-Learning  in  IT  Experts: 

A Phenomenological  Inquiry  Based  on  a Model  of  Expertise  Development 


Andrea  D.  Beesley 
University  of  Oklahoma 

Introduction 

Much  research  has  been  done  on  the  nature  of  expertise,  and  the  differences  between  novices  and 
experts  in  several  domains.  According  to  Anderson  (2000),  skill  acquisition  has  a cognitive  phase  in  which 
people  learn  the  steps  of  a procedure,  an  associative  phase  in  which  the  method  is  worked  out  and  errors  are 
reduced,  and  an  autonomous  phase  in  which  skill  becomes  rapid  and  automatic.  Experts  have  declarative 
knowledge  about  their  domain  proceduralized  for  greater  efficiency  of  performance,  and  rememb  er  information 
about  their  domains  in  chunks  rather  than  as  individual  items.  They  tend  to  reason  forward  from  the  givens  of 
problems  rather  than  backward  from  the  problem  statement,  and  can  focus  on  the  constructs  underlying 
problems  (bottom-up  reasoning)  rather  than  on  surface  features  such  as  knowns  and  unknowns  (top-down 
reasoning).  When  making  decisions,  as  with  chess  players,  for  example  (Horgan,  1988),  experts  do  not  require 
the  exhaustive  evaluations  of  alternatives  that  novices  do,  because  their  experience,  memory,  and  domain- 
specific  knowledge  help  them  narrow  the  range  of  choices  quickly. 

Studies  of  experts  in  different  domains  have  diversified  the  view  of  expertise.  For  example,  expert 
computer  programmers  use  top-down  reasoning,  as  thinking  about  the  breadth  of  their  programs  before  the 
depth  of  each  component  leads  to  better-designed  systems  (Anderson,  2000).  In  a study  of  genetic  counselors 
and  biology  professors,  the  genetic  counselors  focused  more  on  surface  features  such  as  knowns  and  unknowns 
than  did  the  professors,  but  still  outperformed  them  on  genetic  problem-solving  (Smith,  1992).  The  nature  of 
particular  domains,  and  the  context  in  which  skills  are  used  influence  what  expertise  looks  like  in  terms  of 
problem  representation  and  s olution  techniques. 

However,  while  aspects  of  the  nature  of  expertise  are  becoming  known,  the  research  on  explanations 
for  differences  in  people’s  rate  and  level  of  expertise  development  is  sparser  than  that  examining  how  experts 
differ  from  novices  (Alexander,  2003).  Conclusions  range  from  differences  in  deliberate  practice,  talent/ability, 
and  motivation  to  differences  in  metacognition  and  cultural  surroundings. 

Here  I will  review  the  literature  on  the  major  explanations  offered  for  individual  differences  in  the 
development  of  expertise,  propose  a model  of  the  interactions  of  several  factors  to  produce  differences  in 
expertise  development,  and  present  the  results  of  an  initial  qualitative  study  with  information  technology 
experts. 


Deliberate  Practice 

We  may  watch  an  Olympic  athlete  or  concert  pianist  and  marvel  at  the  “talent”  these  individuals 
possess.  Some  researchers,  however,  write  that  the  extraordinary  expertise  of  these  accomplished  performers  is 
not  proof  of  superior  talent  (Sloboda,  1996).  Ericsson  (1996,  2002)  advocated  deliberate  practice,  versus  either 
talent  or  mere  participation,  as  the  explanation  of  differences  in  the  development  of  expertise.  In  his  studies  of 
domains  such  as  music,  sports,  and  dance,  where  competitions  can  help  identify  and  measure  clearly  superior 
performance,  he  concluded  that  expert  performance  is  primarily  acquired  through  many  thousands  of  hours  of 
deliberate  training  and  practice,  because  he  found  direct  positive  correlations  between  hours  of  practice  and 
achievement.  Ericsson  defined  deliberate  practice  as  periods  of  intense  concentration  and  work  that  constantly 
push  the  limits  of  current  capacity,  four  to  five  hours  per  day,  preferably  guided  by  the  best  teachers  or  coaches. 
This  type  of  practice  leads  to  cognitive,  psychological,  and  physiological  changes  that  produce  expert 
performance.  Ericsson  wrote  that  in  order  to  reach  an  international  level  of  competition,  people  need  at  least  10 
years  of  this  type  of  guided  deliberate  practice,  rather  than  special  talent.  Expert  performers  tend  to  start 
practicing  two  to  five  years  earlier  than  less-accomplished  performers,  so  over  time  tend  to  accumulate  more 
hours  of  deliberate  practice  (Ericsson  & Charness,  1997). 

Anderson  (2000)  also  stressed  the  role  of  practice,  writing  that  chess  masters,  for  example,  were  not 
more  generally  intelligent  than  other  people;  they  just  had  practiced  more.  In  his  view,  the  improvement  from 
practice  was  continuous,  if  at  an  ever-diminishing  rate,  and  was  virtually  unlimited  for  cognitive  skills. 


According  to  Ericsson  (2002),  Anderson’s  three-step  model  of  expertise  applies  to  everyday  skills,  but  once 
performance  reaches  the  third  stage  of  automaticity,  one  risks  stagnation.  To  be  truly  expert,  deliberate  practice 
and  specially  designed  training  activities  are  necessary.  In  this  way,  performers  can  develop  skill  but  retain 
cognitive  control  over  aspects  of  the  performance. 

Underlying  deliberate  practice  are  motivation  and  the  acquisition  of  domain -specific  knowledge  that 
can  help  one  monitor  learning  activities  (Ericsson,  1996),  although  Ericsson  did  not  explore  these  factors 
extensively.  Innate  ability  is  a trivial  consideration.  Ericsson  noted  that  athletes  do  not  have  better  simple 
reaction  time,  memory,  or  perception  of  stimuli  than  other  people.  Although  he  acknowledged  that  musicians 
are  more  likely  to  have  perfect  pitch  and  that  better  typists  can  tap  their  fingers  faster,  he  asserted  that  these 
abilities  can  be  skills  acquired  through  training  and  deliberate  practice  rather  than  being  precursors  to  expertise. 

Ericsson  studied  elite  musicians  and  athletes  who  were  guided  intensely  by  coaches  and  teachers. 
Davidson,  Howe,  Moore,  and  Sloboda  (1996),  however,  studied  the  involvement  of  parents  with  young 
musicians  to  see  precisely  how  motivation  and  deliberate  practice  (and  thereby  expert  performance)  could  be 
facilitated  by  parents.  Although  all  the  children  experienced  some  parental  involvement  in  practice,  such  as 
their  listening  to  or  requesting  practice,  successful  children  were  found  to  have  the  most  parental  involvement  in 
lessons  as  well,  across  an  entire  learning  period  of  12-15  years.  When  parents  attended  lessons  and  found  out 
what  the  teachers  were  asking  for,  they  were  better  able  to  guide  practice  at  home.  Davidson  et  al.  (1996) 
acknowledged  that  children  with  greater  talent  might  elicit  greater  parental  involvement,  but  asserted  that  this 
would  not  explain  the  difference  in  rate  of  improvement  over  time,  nor  would  it  increase  parental  involvement 
in  lessons.  They  envisioned  a cycle  in  which  parental  behavior  enhances  achievement,  which  motivates  further 
parental  support,  which  in  turn  enhances  achievement,  and  so  on. 

While  Ericsson  (1996,  2002)  and  Davidson  et  al.  (1996)  studied  experts  whose  development  was 
fostered  by  teachers  and  coaches,  Charness,  Krampe,  and  Mayr  (1996)  studied  chess  champions.  While  they 
found  that  deliberate  practice  carried  the  most  weight  in  determining  skill  level,  they  also  discovered  that 
tutoring  was  relatively  unimportant  in  this  group.  Although  coaches  early  on  might  have  helped  to  provide 
motivation  and  set  up  practice  schedules,  most  champions  studied  and  practiced  alone.  There  was  a significant 
correlation  between  the  number  chess  books  owned  and  achievement  rating.  At  least  in  some  domains,  it  may 
be  possible  to  establish  self-learning  situations  that  facilitate  expertise  development  at  least  as  well  as  teachers 
and  coaches  do. 


Ability  and  Talent 

Sternberg  (1996)  wrote  that  much  of  the  deliberate  practice  research  is  fundamentally  flawed. 
According  to  him,  it  ignores  contradictory  findings,  such  as  work  in  behavior  genetics.  He  also  cited  cases  in 
which  students  who  perform  better  have  worked  less;  for  example,  in  one  of  his  statistics  courses,  students  who 
reported  studying  longer  made  lower  midterm  scores,  which  he  took  to  mean  that  practice  without  ability  does 
not  produce  rewards.  In  addition,  Sternberg  pointed  out  that  studies  of  deliberate  practice  involve  correlation, 
not  causation,  and  do  not  use  control  groups,  making  it  difficult  to  compare  experts  with  those  who  have  had  as 
many  hours  of  deliberate  practice  but  have  not  become  experts.  Deliberate  practice  studies,  according  to 
Sternberg,  ignore  the  fact  that  many  people  who  seek  a high  level  of  expertise  in  a domain  often  drop  out  due  to 
dissatisfaction  with  their  performance,  so  naturally  those  who  remain  display  a correlation  between  expertise 
and  practice.  Finally,  Sternberg  argued  that  deliberate  practice  studies  ignore  common  sense.  What  Mozart 
accomplis  hed  as  a child  is  seldom  matched  by  those  who  accumulate  as  much  practice  as  he  had  had  at  that 
time,  for  example.  Some  graduate  students  are  better  teachers  after  one  semester  than  are  professors  who  have 
taught  for  many  years,  despite  their  inexperience.  Deliberate  practice,  therefore,  is  responsible  for  only  part  of 
expert  performance. 

According  to  Sternberg  (1996),  trying  is  not  enough,  even  when  it  takes  the  form  of  deliberate  practice. 
It  is  likely  that  Winner  (1996)  would  agree.  In  a study  of  children’s  drawings,  she  found  evidence  to  support 
the  existence  of  an  innate  talent  in  the  domain  of  the  visual  arts.  She  found  that  children  who  have  high  ability 
at  a young  age  tend  to  learn  more  rapidly  in  the  domain,  are  intrinsically  motivated  because  of  the  ease  of  their 
learning,  make  artistic  discoveries,  such  as  perspective,  without  scaffolding  from  adults,  and  tend  to  do  creative 
things  that  ordinary  children  do  not  think  to  do.  These  children  tended  to  draw  constantly  without  being  told, 
with  progressively  better  results.  “Regular”  children  may  work  hard,  but  their  progress  over  the  years  was 
much  slower  than  that  of  artistically  gifted  children.  Although  she  agreed  that  hard  work  is  necessary,  she 
argued  that  it  is  not  sufficient,  and  moreover  it  is  difficult  to  separate  hard  work  and  ability.  They  are 
confounded  because  we  are  likely  to  want  to  work  hard  at  something  that  we  can  do  easily.  In  any  case.  Winner 
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found  it  illogical  to  ascribe  mental  retardation  to  a biological  bases  yet  claim  that  high  performance  is  due  only 
to  hard  work. 


Metacognition 

Apart  from  the  debate  over  deliberate  practice  versus  the  role  of  ability,  other  factors  contribute  to 
individual  differences  in  the  development  of  expertise.  Sternberg  (1998)  wrote  that  along  with  abilities, 
metacognition  contributes  to  the  development  of  expertise.  Reasoning  that  metacognition  operates  independent 
of  domain,  intelligence,  and  knowledge  (as  mere  knowledge  does  not  always  lead  to  action),  Veenman  and 
Elshout  (1999)  investigated  the  role  of  metacognition  along  with  intelligence,  to  see  whether  metacognition 
would  contribute  separately  toward  developing  expertise.  They  found  that  the  advanced  students  had  higher 
metacognitive  skillfulness  in  comparison  to  novices  of  both  high  and  low  intelligence  levels.  However,  highly 
intelligent  novices  nearly  reached  the  metacognitive  level  of  advanced  students,  and  tended  to  rapidly  gain 
knowledge  in  the  physics  domain.  For  both  novices  and  advanced  students,  metacognition  contributed  on  its 
own,  apart  from  intelligence. 

However,  very  complex  problems  were  solved  by  the  high-intelligence  students  only.  At  that  high 
level,  metacognitive  skillfulness  was  no  longer  helpful  for  problem  solving.  The  authors  conclude  that  routine 
metacognitive  skills  are  more  important  than  intelligence  for  routine  problems  but  not  for  very  difficult 
problems,  perhaps  because  at  that  point  problems  no  longer  concern  students’  proceduralized  knowledge  but 
rather  require  knowledge  at  the  conceptual  level,  which  higher-intelligence  students  may  be  better  able  to  use. 

Cultural  Factors 

In  considering  how  and  why  experts  are  motivated  to  work  longer  and  harder  than  others,  thus 
acquiring  knowledge,  skills,  and  the  useful  psychological  and  physiological  changes  that  come  from  practice 
(e.g.  Ericsson  1996),  Gleespen  (1996)  sought  other  sources  of  motivation  support  than  the  aforementioned 
parents  and  coaches.  His  cultural  theory  of  expertise  development  states  that  the  cultures  of  experts  tend  to 
support  learning  and  development,  and  are  replete  with  resources  and  opportunities  in  the  relevant  domain. 

This  can  include  teachers  and  parents  where  children  are  concerned,  or  colleagues  and  communities  of  practice 
for  adults.  In  a setting  in  which  experts  and  learners  acknowledge  each  other’s  skills,  collaborate  and  critique 
each  other’s  performance,  and  share  advice,  an  attractive  environment  for  achievement  is  created.  When  people 
are  separated  from  this  environment,  either  by  finding  themselves  in  an  unfamiliar  environment  or  by  returning 
from  a supportive  environment  to  a normative  environment  that  does  not  support  the  domain,  their  performance 
levels  are  likely  to  drop. 

The  idea  of  community  support  leading  to  expertise  in  certain  domains  may  also  apply  to  those  who 
study  relatively  alone,  such  as  the  chess  players  observed  by  Charness,  Krampe,  and  Mayr  (1996).  While  these 
chess  champions  generally  did  not  have  coaches  or  live  in  communities  of  other  chess  players,  they  participated 
regularly  in  tournaments  that  allowed  them  to  receive  feedback  with  which  to  gauge  their  progress,  and  learning 
opportunities  that  may  have  helped  to  refine  their  future  solitary  practice.  In  addition,  with  the  Internet  it  may 
now  be  easier  than  ever  for  people  who  had  pursued  a passbn  in  relative  isolation  to  create  a lively  virtual 
community  that  provides  all  the  expertise-supporting  advantages  mentioned  by  Gleespen  (1996),  and  that  is  less 
sensitive  to  changes  in  a participant’s  physical  location  than  is  the  neighborhood  in  which  one  lives  or  the 
workplace  in  which  one  is  employed.  Now  that  cultural  support  is  more  accessible  and  portable  for  many 
domains,  it  remains  to  be  seen  whether  levels  of  participants’  expertise,  or  the  number  of  existing  experts,  has 
risen. 


Toward  A Multifaceted  Model  of  Differences  in  Expertise 

A unitary  model  (deliberate  practice  only,  talent  only,  for  example)  is  too  simplistic  to  capture  the 
range  of  factors  underlying  differences  in  the  development  of  expertise.  Sternberg  (1999)  proposes  that 
intelligence  is  the  same  as  developing  expertise,  and  while  ability  affects  the  rate  and  asymptote  of  development 
in  a domain,  expertise  has  five  elements:  metacognitive  skills,  learning  skills  (ability  to  distinguish  relevant 
from  irrelevant  information,  for  example),  thinking  skills  (creativity,  critical  thinking,  practical  and  applied 
thinking,  etc.),  declarative  and  procedural  knowledge,  motivation,  and  context  (familiarity  of  material, 
importance  to  the  student,  for  example).  Presumably  individual  differences  in  any  of  these  factors  could  lead  to 
individual  differences  in  expertise  development  itself. 

Sternberg’s  multifactorial  explanation  of  expertise  is  welcome  in  the  face  of  overly  simplified  models. 
However,  it  does  not  explain  how  these  elements  maybe  related,  and  does  not  elaborate  on  the  motivation 
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component.  That  motivation  is  required  on  the  path  to  expertise  is  unsurprising;  those  with  no  motivation  for 
developing  expertise  are  unlikely  to  persist  in  the  domain.  However,  what  form  this  motivation  may  take  has 
been  largely  unexplored  in  studies  of  expertise.  Despite  the  paucity  of  research,  the  theories  of  achievement 
goals,  self-efficacy,  and  flow  suggest  connections  to  other  factors  in  expertise  development. 

Achievement  goals 

People  often  describe  developing  experts  as  “goal-oriented.”  In  Dweck’s  study  of  adaptive  and 
maladaptive  motivational  patterns  (1986),  she  described  goal-oriented  activity  with  two  factors,  learning  and 
performance  goals.  Individuals  with  learning  goals  “seek  to  increase  their  competence,  to  understand  or  master 
something  new,”  while  those  with  performance  goals  “seek  to  gain  favorable  judgments  of  their  competence” 

(p.  1040).  In  their  work,  Elliott  and  his  colleagues  have  further  divided  performance  goals  into  performance- 
approach  and  performance-avoidance  goals.  Those  with  performance-approach  goals  wish  to  do  better  than 
others,  while  those  with  performance-avoidance  strive  to  avoid  doing  worse  than  others  (Elliott  & Church, 

1997;  Elliott  & Thrash,  2001).  Performance-approach  goal  regulation  can  include  either  a need  for 
achievement,  in  which  people  eagerly  approach  the  task,  or  a fear  of  failure,  in  which  people  approach  the  task 
and  work  very  hard  (overstrive)  because  they  do  not  want  to  fail  (Harackeiwicz,  Barron,  Pintrich,  Elliott,  & 
Thrash,  2002b).  Performance-avoidant  people  try  to  escape  the  achievement  situation  altogether,  or  as  Dweck 
(1986)  also  described,  choose  very  easy  or  very  difficult  tasks. 

Because  achievement  goal  studies  often  focus  on  subjects  who  are  not  necessarily  experts,  it  is  difficult 
to  hypothesize  about  the  specific  goals  of  experts  by  relying  on  previous  research.  However,  the  expectation  of 
high  self-efficacy  in  experts,  and  the  fact  that  experts  are  people  who  perform  well  on  challenging  domain - 
related  tasks,  imply  that  experts  are  likely  to  hold  learning  goals  as  well  as  performance-approach  goals.  Recent 
research  addresses  the  possibility  that  people  may  not  be  dominated  by  either  learning  or  performance  goals  but 
may  endorse  multiple  goals,  and  that  a combination  of  learning  and  performance-approach  goals  is  beneficial  to 
achievement.  Harackiewicz,  Barron,  Tauer,  Carter,  and  Elliot  (2000)  studied  introductory  psychology  college 
students  over  three  semesters,  and  found  that  performance-approach  goals  predicted  short-  and  long-term 
academic  performance  and  that  mastery  goals  predicted  short-term  interest  in  the  course  and  enrollment  in 
subsequen  t psychology  courses.  Therefore  they  suggested  that  optimal  goal  adoption  consist  of  both 
performance-approach  and  mastery  goals,  because  both  grade  performance  and  continued  interest  are 
important  to  success  in  college.  When  the  researchers  followed  some  of  these  students  to  graduation 
(Harackiewicz,  Barron,  Tauer,  & Elliot,  2002a),  they  found  the  same  predictive  pattern,  although  the  effects  of 
performance  goals  weakened  over  time. 

Accepting  that  practice  is  one  factor  leading  to  expertise,  and  that  deliberate  practice  involves  intense 
concentration  and  continuous  striving  for  progress,  it  would  seem  that  learning/mastery  goals  would  be 
effective  in  expertise  development,  since  learning  goals  tend  to  foster  persistence,  effort,  and  improvement 
(Dweck,  1996).  Because  expertise  requires  years  of  involvement  in  a domain,  the  interest-sustaining  properties 
of  learning  goals  ( Harackiewicz  et  al.,  2000,  2002a)  would  also  be  important.  However,  many  expertise 
domains,  such  as  music  and  athletics,  are  dominated  by  the  need  to  perform  for  others,  and  even  in  settings  such 
as  the  workplace  one  must  be  concerned  with  performance  and  others’  judgments  of  one’s  ability.  It  is  also 
possible  that  people  with  little  inclination  to  demonstrate  their  competence  to  others  would  not  become  known 
as  experts.  Therefore,  those  who  have  been  successful  in  developing  expertise  are  likely  to  have  both  learning 
goals  and  performance-approach  goals  to  some  degree,  similar  to  the  successful  college  students  in 
Harackiewicz  et  al.  (2000,  2002a). 


Self-efficacy 

Perceived  ability,  also  known  as  self-efficacy  (Bandura,  1994),  interacts  with  achievement  goals,  as 
mentioned  above.  When  perceived  ability  is  high,  people  tend  to  persist  when  challenged  regardless  of  goal 
type,  but  when  perceived  ability  is  low,  they  give  up  quickly  if  driven  by  performance  goals,  which  is 
incompatible  with  the  development  of  expertise.  High  self-efficacy  also  leads  people  to  set  higher  goals,  which 
in  turn  raises  the  level  of  motivation  for  and  attainment  of  the  task  (Bandura,  1989),  all  of  which  would 
contribute  to  expert  performance. 

However,  self-efficacy  also  affects  expert  performance  directly  (Bandura,  1989,  1994).  In  a difficult 
performance  situation,  people  with  high  self-efficacy  believe  that  they  can  cope  with  challenges,  and  therefore 
are  not  unduly  bothered  by  them  and  can  remain  task-oriented.  However,  people  with  low  self-efficacy  become 
consumed  with  self-doubt  during  a challenging  performance.  The  self-doubt  is  distracting,  causing  anxiety  and 
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interfering  with  thinking  and  concentration,  thus  leading  to  a lower  quality  of  performance.  The  lower  quality 
of  performance  then  leads  to  lower  self-efficacy  for  the  future,  and  the  cycle  continues.  In  the  road  to  expertise, 
then,  self-efficacy  exerts  an  influence  on  motivation  by  interacting  with  goals,  but  also  has  a direct  connection 
to  task  performance  itself. 


Flow 

Given  the  importance  of  deliberate  practice  as  a component  of  expertise  development,  and  given  that 
deliberate  practice  is  characterized  by  intense  concentration  that  pushes  the  boundaries  of  capacity,  it  is  possible 
that  individuals  engaged  in  deliberate  practice  experience  flow  (Csikszentmihalyi,  1990).  Individuals  in  flow 
are  in  a self-chosen  and  optimally  challenging  environment  in  which  their  ability  and  the  task  challenges  are 
well  matched.  The  task  is  difficult  enough  to  require  some  skill,  and  the  performer  is  no  longer  a complete 
novice  (or  her  skill  level  would  be  too  low  to  be  able  to  perform  well  enough  to  achieve  flow).  Self-efficacy  is 
high,  so  intrusive  feelings  of  self-doubt  do  not  interrupt  the  performance  of  the  task.  The  task  has  a clear  goal, 
and  the  task  environment  provides  feedback  that  is  comprehensible  to  the  performer  and  to  which  he  is  able  to 
react  appropriately.  The  intense  concentration  drives  out  distracting  extraneous  thoughts  and  worries,  and  leads 
the  performer  to  lose  track  of  time.  In  this  state  the  performer  has  a sense  of  control  and  competence,  but  is  still 
challenged.  While  during  flow  the  performer  loses  the  habitual  sense  of  self-consciousness,  afterward  people 
often  feel  a stronger  sense  of  self.  At  the  time  the  flow  experience  may  be  too  intense  to  feel  pleasurable  (or 
people  are  too  engaged  in  it  to  stop  and  think  about  whether  it  is  pleasurable),  but  later  one  looks  back  on  it  with 
happiness,  accomplishment,  and  satisfaction,  which  is  rarely  the  case  with  more  relaxing  pursuits.  In  the 
context  of  expertise,  flow  and  deliberate  practice  may  sometimes  be  one  and  the  same,  or  flow  may  be  a 
particularly  rewarding  component  of  practice  and  performance  that  in  turn  motivates  further  practice. 

A Model  of  Factors  Influencing  the  Development  of  Expertise 

In  Figure  1,  “Factors  Influencing  the  Development  of  Expertise,”  I propose  a set  of  relationships 
between  aspects  of  expert  performance  described  in  the  literature  summarized  above,  such  as  deliberate 
practice,  motivation,  support  from  teachers  and  coaches,  support  from  culture  and  communities  of  practice, 
metacognition,  domain  knowledge,  and  talent/intelligence.  I also  include  factors  not  previously  linked 
explicitly  to  individual  differences  in  the  development  of  expertise,  such  as  goals,  self-efficacy,  and  flow. 

Figure  1.  Factors  Influencing  the  Development  of  Expertise 
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Description  of  the  Model 

In  the  model,  four  primary  elements  lead  to  motivation:  support  from  teachers,  parents,  and/or  coaches; 
support  from  one’s  culture  or  community  of  practice;  goals;  and  values.  Support  from  teachers,  parents  and 
coaches  would  tend  to  be  individualized  and  more  intense  than  that  from  culture  and  communities  of  practice, 
but  both  serve  to  add  to  domain  knowledge  and  sustain  motivation.  Having  specific  goals,  perhaps  especially 
learning  goals,  also  contributes  to  motivation,  as  does  placing  value  on  the  goal  or  domain.  Motivation  in  turn 
sustains  the  work  of  deliberate  practice,  necessary  for  expert  achievement.  Deliberate  practice  may  include 
periods  of  flow  that  sustain  motivation  to  engage  in  further  practice.  In  the  model,  ability/  intelligence/talent 
helps  expert  performance  develop  faster  and  farther.  It  also  aids  in  the  faster  accrual  of  domain  knowledge,  and 
supports  flow,  because  a fairly  high  level  of  competence  is  necessary  to  achieve  flow.  Domain  knowledge  in 
turn  contributes  to  the  effectiveness  of  deliberate  practice  and  in  turn  is  increased  by  it,  and  also  contributes  to 
expert  performance  directly.  Metacognition  helps  ensure  the  quality  of  deliberate  practice  as  well  as  contributes 
independently  to  expert  performance.  Finally,  expert  performance  starts  the  cycle  again,  as  it  tends  to  foster 
further  support  from  teachers  and  the  like,  along  with  cultures  and  communities  of  practice  to  whom  one’s 
accomplishments  become  known.  It  also  validates  and  strengthens  the  original  goals  and  values  that  led  to 
motivation  in  the  first  place. 

This  preliminary  model  can  help  to  illustrate  some  of  the  ways  in  which  individuals  may  differ  from 
one  another  in  the  development  of  expertise.  A failure  in  any  node  of  the  structure  has  the  potential  to  imperil 
or  delay  expertise  development.  For  example,  lack  of  support  from  one’s  culture  or  lack  of  proper  instruction 
diminishes  motivation  and  domain  knowledge.  Failure  to  engage  in  deliberate  practice  detracts  from 
performance,  but  also  fails  to  develop  domain  knowledge.  Low  talent  or  intelligence  puts  domain  knowledge, 
the  possibility  of  flow,  and  expert  performance  at  risk.  Just  as  expert  performance  is  the  result  of  many  related 
factors,  differences  in  the  achievement  of  expertise  can  be  due  to  one  or  more  of  several  causes. 

Rationale  for  a Qualitative  Study 

Pursuing  a new,  complex  model  such  as  “Factors  Influencing  the  Development  of  Expertise’’  with 
quantitative  methods  first  may  be  presuming  too  much  about  the  experiences  of  experts.  Therefore,  I conducted 


93 


an  initial  qualitative  study  of  a group  of  experts  in  order  to  have  a better  understanding  of  if  and  how  the 
hypothesized  elements  in  the  model  exist  in  people  developing  expertise. 

Although  the  nature  and  development  of  expertise  in  general  have  been  widely  studied,  little  work  has 
been  done  on  the  motivational  characteristics  of  developing  experts,  or  on  those  who  teach  themselves  to  the 
level  of  expertise.  Since  there  remains  much  to  be  learned  about  those  who  self-teach,  a qualitative  approach 
would  allow  for  a full  exploration  of  the  phenomenon  by  allowing  those  who  do  it  to  explain  and  describe  it  as 
they  experience  it.  A qualitative  study  may  help  identify  categories  and  themes  that  will  allow  for  further,  more 
narrowly  -focused  quantitative  work. 


Purpose 

For  this  purpose,  any  group  of  experts  would  arguably  be  appropriate.  At  least  in  initial  studies, 
however,  the  domain  of  expertise  should  be  the  same  or  similar  for  all  members  of  the  subject  group,  because 
the  characteristics  of  the  domain  itself  may  influence  the  nature  of  its  experts  so  much  that  the  results  of 
qualitative  studies  with  experts  from  multiple  domains  may  be  difficult  to  interpret.  I chose  self-taught 
information  technology  (IT)  experts  for  this  study.  Information  technology  is  a complex  and  varied  field  that  is 
constantly  changing.  Experts  in  this  field  must  acquire  and  continuously  update  a large  amount  of  domain 
knowledge,  such  as  network  structures  and  programming  languages,  and  be  able  to  use  it  effectively.  Although 
the  field  is  demanding  and  often  stressful,  information  technology  products  and  people  are  present  throughout 
the  economy  (Hilton,  2001),  so  this  domain  of  expertise  is  less  exclusive  than  that  of  the  elite  athletes  and 
musicians  studied  by  other  expertise  researchers. 

Unlike  many  other  professions,  the  field  of  information  technology  is  made  up  of  people  who  come 
from  a variety  of  educational  backgrounds.  Employers  are  more  interested  in  the  hands-on  experience  and 
technical  knowledge  of  potential  hires  than  in  their  degrees.  One-third  of  IT  workers  have  only  a high  school 
diploma  or  two-year  degree,  and  of  the  two  -thirds  that  have  at  least  a bachelor’s  degree,  less  than  half  have  a 
major  or  minor  in  a computer-related  field  (Hilton,  2001).  Therefore,  most  IT  workers  have  come  by  their 
knowledge  from  informal  learning  or  self -teaching,  similar  to  the  chess  players  studied  by  Charness,  Krampe, 
and  Mayr  (1996).  Achieving  expertise  is  effortful  in  all  domains,  but  self-taught  IT  workers  seem  to  have 
accomplished  it  with  less  formal  support  (teachers,  coaches,  degree  programs)  than  have  experts  in  areas  where 
the  path  to  success  is  less  self-determined.  Therefore  self-taught  IT  experts  might  be  exemplary  in  their  ability 
to  forge  their  own  path  to  expertise,  and  their  techniques  illuminating  to  others  who  are  attempting  to  do  so,  or 
to  figure  out  how  expertise  development  happens. 

The  purpose  of  this  initial  qualitative  study  is  to  explore  the  self -teaching  strategies  and  behaviors  of 
developing  IT  experts,  and  to  see  whether  there  is  support  in  the  data  for  the  factors  included  in  the  expertise 
development  model.  The  central  questions  of  the  study  are: 


1.  What  are  the  characteristics  of  self-taught  IT  workers? 

2.  How  do  those  who  become  experts  on  their  own  find  and  utilize 
appropriate  resources  to  support  their  development? 

3.  How  do  they  measure  their  progress? 

4.  What  sustains  their  motivation? 


Since  many  other  workers  today  must  learn  and  adapt  to  new  technology  quickly,  often  without 
formal  training,  the  strategies  of  self-taught  IT  experts  could  help  others  learn  technology  easier  and  more 
thoroughly. 


Phenomenological  Approach 

A phenomenological  approach  describes  the  meaning  of  the  experiences  of  a group  of  individuals  who 
have  undergone  the  same  phenomenon.  Its  roots  are  in  the  philosophy  of  Husserl,  Heidegger,  Saitre,  and 
Merleau-Ponty.  Phenomenological  study  involves  the  search  for  the  essence  of  the  meaning  of  an  experienced 
phenomenon.  Data  analysis  in  phenomenology  aims  to  reduce  the  information  into  a set  of  themes  (Creswell, 
1998).  Those  who  are  participants  in  a phenomenological  study  must  have  experienced  the  phenomenon  to  be 
in  the  study  group.  Researchers  allow  the  participants  to  explain  their  experience  of  the  phenomenon  from  their 
own  perspectives  and  in  their  own  voices. 
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Sampling 

Because  a phenomenological  approach  requires  participants  to  have  experienced  the  phenomenon  of 
interest,  criterion  sampling  was  used.  In  this  study,  the  criterion  was  met  if  participants  answered  “yes”  to  both 
of  the  following  questions:  “Are  you  currently  working  in  the  information  technology  / computer  science  field? 
Is  the  knowledge  of  information  technology  that  you  are  using  in  your  work  derived  mainly  from  self-study  (as 
opposed  to  formal  university  coursework  or  training  classes)?” 

However,  another  qualification  is  that  the  workers  have  to  be  well  along  the  path  to  expertise.  Testing 
the  IT  knowledge  of  the  interviewees  with  some  sort  of  task  was  impractical,  not  only  because  it  would  have 
been  overly  demanding  of  the  time  of  the  subjects  but  also  because  the  IT  workers  can  come  from  various 
subfields  which  may  not  share  all  of  the  same  content  (although  according  to  Hilton  (2001)  they  do  share 
roughly  the  same  percentage  of  informal  learners).  The  best-performing  IT  workers,  however,  build 
communications  with  other  IT  colleagues  and  spend  a lot  of  time  gathering  and  sharing  information  (Hilton, 
2001).  IT  work  also  has  results  that  are  easily  observable  by  others — whether  the  application  works,  whether 
the  database  is  useful,  whether  or  not  the  servers  run  well — so  workers’  level  of  performance  is  not  easily 
hidden,  especially  from  other  IT  professionals.  Through  these  methods  IT  workers  can  come  to  know  the  skill 
levels  of  others.  Therefore  a snowball  sampling  technique,  in  which  knowledgeable  people  recommend  other 
cases  who  in  turn  recommend  other  cases,  and  so  on  (Gall,  Gall,  & Borg,  2003),  was  appropriate  in  conjunction 
with  the  qualifying  questions.  The  initial  cases  were  experienced  IT  managers  known  to  the  researcher.  Six 
people  were  interviewed  for  this  study. 


Data  Collection 

Data  collection  included  audiotaping  interviews  conducted  questions  derived  from  the  elements  of  the 
model  “Factors  Influencing  the  Development  of  Expertise.”  An  initial  interview  was  conducted  with  each 
participant,  and  followed  up  and  verified  as  needed  while  data  analysis  continued.  To  ensure  confidentiality, 
each  participant  chose  a code  name;  in  the  results  all  participants  are  indicated  by  their  code  names.  Other 
forms  of  qualitative  data  collection,  such  as  observation,  were  less  helpful  in  this  specific  study.  When  a learner 
sits  at  the  computer,  perhaps  with  a manual  or  some  other  study  material,  there  is  little  in  the  way  of 
conversation  or  interaction  to  observe.  In  future  studies,  however,  conversations  between  IT  woikers,  chat  logs, 
or  discussion  board  postings  could  be  useful. 


Data  Analysis 

The  modification  of  the  Stevick-Colaizzi-Keen  method  for  analyzing  qualitative  data  in  the 
phenomenological  tradition,  as  described  in  Creswell  (1998),  was  the  model  for  handling  the  qualitative  data 
collected.  Beginning  with  transcripts  from  audiotaped  interviews  with  the  subjects,  I listed  out  significant 
statements  (sentences,  clauses,  or  phrases)  about  how  individuals  experienced  the  phenomenon,  giving  them 
equal  worth.  Using  statements  instead  of  codes  allowed  for  longer  units  of  meaning,  since  some  of  the 
participants’  thoughts  cannot  be  adequately  summed  up  in  a word  or  two.  I eliminated  statements  that  are  not 
relevant  to  the  topic  of  interest,  and  removed  or  combined  statements  that  overlap  with  other  statements.  I 
grouped  the  statements  into  clusters,  or  categories,  of  meaning,  and  displayed  them  in  tables.  Finally  I 
constructed  a blended  description  of  the  overall  meaning  of  people’s  experiences,  and  then  took  the  information 
back  to  the  participants  for  verification. 


Results 

The  following  tables  list  some  significant  statements  from  interviewees  on  the  categories  that  emerged 
from  the  data.  Each  category  is  labeled  with  its  title  and  followed,  in  parentheses,  by  the  element  of  the 
expertise  model  to  which  the  category  relates,  or  the  topic  of  the  research  question  addressed.  Each  table  is 
followed  by  a brief  composite  description. 

Table  1.  Curious  about  Technology  Early  On  (Talent/ Ability,  Research  Question  1:  Characteristics) 

Subject  code  Statement 

name 

Bigglesworth  “Ever  since  I was  a kid  I pulled  knobs  off  of  things,  pulled  things  apart.” 

“I  got  curious  about  the  operating  system  and  I broke  our  brand-new  computer.” 

Skippy  “When  I was  in  5th  or  6th  grade  in  elementary  school. ..that  was  my  first  system 

administration  gig.” 
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“They  shipped  them  with  books,  and  I read  them  and  played  around.... Reading  the 
manuals,  and  error  and  error.” 

Bub  “I  always  enjoyed  building  things,  putting  things  together  and  seeing  how  they  worked.” 

Robotech  “I  started  taking  programming  classes  when  I was  in  the  eighth  grade.  I just  kind  of 

steered  towards  it.  It  was  just  a natural  fit.” 

“I’ve  always  been  real  gadgety,  taking  stereos  apart,  things  like  that.” 

The  subjects  reported  being  interested  in  taking  things  apart  to  see  how  they  worked.  They  often 
began  using  computers  in  elementary  school.  Instead  of  merely  running  programs  on  them  or  playing  with 
them,  they  examined  their  inner  system  structures,  sometimes  with  unexpected  results,  and  started  programming 
them. 

Table  2.  Current  Duties  (Research  Question  1:  Characteristics) 

Subject  code  Statement 

name 

Bigglesworth  “I  am  an  IT  Analyst.  My  primary  duties — I am  lead  programmer,  so  I am  also  assisting 

the  other  coders  in  my  department,  helping  with  direction  and  everything  like  that.  My 
primary  task  is  just  to  develop  applications.” 

Skippy  “I’m  the  alpha  geek.” 

“I  keep  the  machines  running  right,  I write  a bit  of  code  here  and  there.  I think  my  main 
responsibility  is  as  the  fallback  guy  in  the  office.” 

Bub  “It’s  largely  user  support.  There’s  also  network  maintenance  and  some  [database]  design.” 

Robotech  “We  do  all  the  technical  manuals.. ..We’ve  got  our  network  servers,  work  stations,  about  25 

users. ..Right  now  I’m  developing  three  training  courses. ..I’m  a system  administrator.” 

Elroy  “Mostly  administrivia,  much  to  my  chagrin.” 

“Every  chance  I get.. I’m  either  doing  systems  work,  which  is  my  home,  or  I tend  to  try  to 
work  on  database  and  web  applications  of  one  kind  of  another.” 

Trogdor  “Systems  administration. ..secondly,  programming,  because  that’s  really  what  I enjoy  most 

of  all.” 

The  IT  workers  came  from  different  areas  of  the  field,  and  had  multiple  duties.  The  current 
responsibilities  of  the  IT  workers  include  programming  and  systems  administration,  user  support  and  training, 
database  design,  administration,  and  serving  as  a resource  to  less  experienced  workers. 

Table  3.  Enjoying  Optimal  Challenge,  Learning  (Flow,  Research  Question  4:  Sustaining  Motivation) 

Subject  code  name  Statement 

Bigglesworth  “It’s  not  so  challenging  it  just  drives  you  insane,  but  it’s  not  so  easy  that  you’re  bored.” 

“When  you  conquer  it  you  go,  ‘Cool,  now  I know  how  to  do  that.’” 

“It  keeps  you  right  on  the  edge.” 

Skippy  “Solving  problems  and  making  lights  turn  green,  that’s  where  it’s  at.” 

“Not  knowing  how  stuff  works  itches.” 

Bub  “Finding  a solution  is  definitely  a major  relief.” 

Robotech  “I’ve  conquered  it.  That’s  my  biggest  feeling.  I’ve  defeated  it.  I’m  the  master  of  the 

world.” 

Elroy  “A  different  manifestation  of  the  theory  of  relativity. ...It  literally  seems  like  minutes  and  it’s 

been  hours.  It  applies  to  the  high  and  low  moments,  the  breakthrough  and  really  frustrating 
moments.  They  both  pass  as  part  of  one  big  continuum  ....” 

“I  don’t  have  any  trouble  blocking  out  distractions.” 

“...Being  happy  about  getting  my  mind  around  it.” 

Trogdor  “I  get  something  to  work,  and  it’s  like,  ‘Woooh,  yay!’  I probably  get  up  and  dance  around.” 

The  participants  enjoy  the  challenge  of  problemsolving  and  learning  how  new  technology  works. 

When  they  solve  a problem  or  figure  something  out,  they  experience  feelings  of  happiness,  satisfaction,  or 
relief.  Some  describe  a state  of  optimal  challenge,  where  they  are  working  at  the  edge  of  their  skill  level  and 
the  task  is  neither  impossibly  difficult  nor  too  easy,  which  is  consistent  with  the  concept  of  flow. 
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Table  4.  Sense  of  Focus  while  Learning  and  Working  (Flow,  Research  Question  4:  Sustaining  Motivation ) 


Subject  code  name 

Statement 

Bigglesworth 

“I’d  come  into  work  at  8 and  wouldn’t  leave  until  8,  learning  it.” 
“I  put  headsets  on  and  listen  to  music.” 

Skippy 

“It’s  called  larval  stage.  You  just  kind  of  shut  out  everything  and  turn  completely  inside,  and 
you  re-emerge  later.” 

“I  don’t  hear  people’s  voices  and  I don’t  hear  the  phone.  I certainly  don’t  see  the  clock.” 

“On  some  level  your  mind  is  trying  to  protect  your  process.” 

Bub 

“It  can  happen.  It  doesn’t  happen  often,  just  because  of  the  nature  of  this  job.  I wear  so  many 
hats.” 

Robotech 

“I  get  there  about  6:15  in  the  morning,  and  the  next  thing  I know  is  it’s  already  2:00,  and  I 
leave  at  about  10  after  3:00.” 

Elroy 

“A  different  manifestation  of  the  theory  of  relativity. ...It  literally  seems  like  minutes  and  it’s 
been  hours.  It  applies  to  the  high  and  low  moments,  the  breakthrough  and  really  frustrating 
moments.  They  both  pass  as  part  of  one  big  continuum  ....” 

“I  don’t  have  any  trouble  blocking  out  distractions.” 

Trogdor 

“If  the  circumstances  are  right,  I can  definitely  get  sucked  in,  and  get  in  the  whole  flow 
experience,  however  you  pronounce  that  guy’s  name  that  wrote  that  book...  In  fact  I’m  very 
much  influenced  by  that  goal  in  choosing  what  I try  to  do.” 

The  interviewees  are  capable  of  intense  focus  while  learning  new  information  technology — what  Skippy  called 
“larval  stage.”  They  block  out  external  distractions,  such  as  others’  voices,  and  sometimes  lose  the  sense  of  the 
passage  of  time.  Trogdor  (unprompted  by  the  researcher)  even  used  the  word  “flow”  to  describe  his 
experiences,  and  mentioned  that  it  was  a motivator  in  choosing  tasks.  However,  the  nature  of  the  job  can 
interfere  with  this  sense  of  focus;  Bub  reported  that  he  is  constantly  getting  interrupted. 

Table  5.  Making  Connections  to  Prior  Knowledge  ( Metacognition , Domain  Knowledge,  Research  Question  1: 
Characteristics  ) 

Subject  code  name  Statement 

Bigglesworth  “If  you’ve  learned  one  language  sometimes  it  translates  into  another.  I was  able  to  translate 

some  of  the  knowledge  I had  previously  in  C to  Perl.” 

Skippy  “It’s  as  much  of  a shift  between  the  old  version  and  the  new  version  as  between  Windows 

NT  and  2000.” 

Bub  “The  basics  are  always  there,  it’s  just  new  applications  for  them.” 

Robotech  “If  it’s  a Windows-based  product  or  an  Apple  product,  I can  pick  it  up  within  a couple  of 

months.” 


The  interviewees  described  how  they  used  prior  knowledge  to  help  with  the  new  learning  task.  In 
Skippy’ s case,  he  had  experience  shifting  from  one  server  version  to  another,  and  had  used  some  of  the 
components  of  the  current  system,  such  as  Active  Directory.  Bigglesworth,  who  was  learning  a new 
programming  language,  found  connections  between  it  and  a language  he  had  previously  mastered.  Bub  based 
his  approach  to  computer  security  on  the  basics  of  how  computers  work,  and  Robotech  used  his  knowledge  of 
operating  systems  to  understand  new  applications. 


Table  6.  Finding  Resources  with  Help  from  Others  ( Culture , Research  Question  2:  Finding  Resources) 


Subject  code  name 

Bigglesworth 

Skippy 

Bub 

Robotech 

Elroy 

Trogdor 


Statement 

“[The  book]  was  by  recommendation,  a lot  of  my  friends  that  I know  who  are  in  IT.” 
“Geeks  talk  about  books  a lot.” 

“There  are  a number  of  web  sites  that  I check  on  a regular  basis.... We  also  have  a mailing 
list  here. ..of  IT  people,  called  Tech  Support.” 

“The  course  that  they  sent  me  to  was  helpful.” 

“If  I had  seen  someone  swear  by  a book  I would  just  order  it  or  buy  it.” 

“After  Amazon.com,  I’d  say  wait,  let  me  check  that  book  out.  And  everybody  trashed  that 
book.  Aha!  OK,  it  wasn’t  me,  it  was  the  book.” 
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The  interviewees  got  information  from  a variety  of  sources  about  which  resources  to  use  when  learning 
new  technology.  When  the  technology  is  new,  only  the  manufacturer’s  documentation  is  available.  However, 
when  there  are  multiple  sources,  information  can  be  obtained  from  fellow  “geeks,”  either  in  person,  through  the 
World  Wide  Web,  through  mailing  lists,  through  e-mail,  or  on  Usenet.  Both  interviewees  mentioned  books  as 
an  important  source,  and  that  other  IT  people  make  recommendations  about  good  books.  Trogdor  followed  the 
reviews  on  Amazon.com,  while  Robotech  said  that  they  vary  too  much  to  be  useful. 


Table  7.  Learning  Activities  ( Deliberate  Practice,  Research  Question  2:  Using  Resources) 


Subject  code  name 

Statement 

Bigglesworth 

“I  came  up  with  problems  for  myself.” 

Skippy 

“The  first  thing  I did  was  read  the  documentation  front  to  back  at  a sitting.” 
“You  think  of  a configuration,  ‘Can  I get  it  to  do  this?”’ 

Bub 

“I  will  experiment  with  ideas  at  home  on  my  smaller  systems  that  might  be  translatable  to 
the  larger  systems  here.” 

Robotech 

“Usually  I sat  there  and  hacked  away  at  it.” 

Elroy 

“Many,  many,  many  nights  of  just  staying  in  the  server  room  until  it  got  resolved  in  some 
way.” 

Trogdor 

“Assembly  language  is  the  hardest  language,  except  for  maybe  direct  machine  code. ..so  if  I 
could  write  in  assembly  language,  then  I was  a real  programmer.” 

When  setting  out  to  learn  new  software  or  a new  language,  reading  the  documentation  was  an 
important  first  step.  Then  the  learners  moved  to  the  keyboard  to  apply  the  knowledge,  using  problems  or 
scenarios  they  made  up  that  either  teach  general  principals  or  resemble  situations  in  the  workplace.  When 
difficulties  are  encountered,  they  often  stay  working  on  the  problem  for  a long  period  of  time  until  it  is  resolved, 
sometimes  referring  to  documentation  or  the  Internet  for  advice. 


Table  8.  Measuring  Progress  (Metacognition,  Research  Question  3:  Measuring  Progress) 


Subject  code  name 
Biggies  worth 
Skippy 
Bub 

Robotech 


Elroy 


Statement 

“I  usually  go  back  and  look  at  what  I’ve  done  before.” 

“While  learning,  keep  checking,  ‘OK,  now  what  am  I missing?”’ 

“The  fact  that  we  haven’t  had  a major  attack  in  a long  time  is  probably  the  best  indicator.” 
“Hopefully  I have  people  henpecking  me  a lot,  because  if  they  do  it  means  they  trust  me  to 
help  them  fix  whatever  it  is. ...If  I go  in  there  in  the  morning  and  not  get  badgered  every  30 
minutes,  I know  something’s  wrong.  Or  everything’s  working  really  well.” 

“Most  of  the  time,  my  thought  is  really,  am  I enjoying  it  or  not.  And  as  long  as  it’s  yes,  I 
keep  going.” 


With  these  IT  workers,  the  indicator  of  progress  was  idiosyncratic.  For  Skippy,  measuring  progress  with  a new 
software  package  included  repeatedly  going  through  the  options  to  see  what  has  yet  to  be  learned.  For 
Bigglesworth,  learning  a new  language  involved  looking  at  one’s  old  code  with  more  experienced  eyes,  to 
detect  differences  between  what  was  done  in  the  past  and  how  much  better  it  could  be  done  now.  For  Bub,  the 
absence  of  attacks  on  the  system  indicated  that  he  was  keeping  up  with  security  information.  Robotech  gauged 
his  knowledge  by  the  amount  of  questions  other  workers  asked  him.  Elroy  used  a feeling  of  enjoyment  as  a 
barometer  of  how  much  he  was  learning. 


Table  9.  Reluctance  to  Ask  Questions  (Culture,  Research  Question  1:  Characteristics) 


Subject  code  name 

Statement 

Bigglesworth 

“I’ve  worked  on  the  same  problem  for  about  two  weeks  before  I’d  finally  given  up  and 
asked.” 

“If  someone  tells  me  something  it  kind  of  goes  in  one  ear  and  out  the  other  a lot  of  times.” 
“I  feel  like  in  order  for  me  to  learn,  I have  to  actually  sit  down  and  go  through  the  process 
for  me  to  be  getting  it.” 

Skippy 

“The  harder  the  question,  the  less  likely  it  is  that  somebody  else  will  know  the  answer  off 
the  top  of  their  head.” 

“You’d  better  be  damn  sure  it’s  a good  question.” 
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Robotech 


“It’s  about  the  worst  thing  you  can  do  in  the  world,  to  waste  a geek’s  time.” 

“It’s  easier  to  go  ask  somebody  and  have  them  show  you  than  it  is  to  spend  hours  looking 
the  answer  up.” 

“I  only  want  to  ask  them  if  it’s  absolutely  necessary.  If  I’m  going  to  do  something  that  I 
know  will  cause  problems  if  I do  it  wrong,  I might  ask  first.” 


While  learning,  some  of  the  IT  workers  tried  to  figure  things  out  for  themselves  using  sources  such  as 
books  and  the  Internet,  rather  than  asking  someone  a question.  Even  when  the  answer  was  not  found  quickly, 
they  continued  to  search  on  their  own — up  to  two  weeks  for  Bigglesworth.  The  reasons  for  this  reluctance  to 
ask  questions  include  the  perception  that  the  question  is  too  difficult  for  someone  else  to  answer  easily,  the 
desire  not  to  waste  another’s  time,  the  belief  that  getting  answers  from  others  diminishes  one’s  own  learning, 
and  the  satisfaction  resulting  from  figuring  it  out  oneself.  Robotech  was  more  willing  to  ask  questions,  but  only 
when  the  task  was  important  and  the  time  savings  was  significant. 


Table  10.  Goals  for  Self-Learning  ( Goals,  Research  Question  4:  Sustaining  Motivation ) 


Subject  code  name 

Statement 

Bigglesworth 

“Learning  it  because  I needed  to  for  a project  or  a problem  that  I needed  to  solve,  that’s 
definitely  a big  factor.” 

“I  also  went  above  the  knowledge  I needed  to  learn.  I went  and  learned  more  simply 
because  you  know  it  may  come  in  handy  later.” 

“I  enjoyed  it.” 

“But  it’s  just  one  of  those  things  that  stimulates  you  to  go  and  try  to  do  more  because 
you  know  you’ve  . . . got  to  show  off  your  skills  in  front  of  somebody  else.” 

Skippy 

“There’s  the  utility  of  it,  if  you  have  this  you  can  do  this.” 

“There’s  a little  ego  thing,  being  the  guy  in  the  office  who  can  say  yeah,  actually  that’s 
done  this  way.” 

“Mostly  it’s  just  fun  digging  around.  It’s  a video  game,  it’s  a new  toy  at  Christmas,  take 
it  apart,  see  how  it  works.” 

Bub 

“It’s  keeping  the  information  stores  here  safe  that  are  entrusted  to  me,  and  making  the 
computer  and  information  resources  available  to  the  people  that  I serve  without 
interruption.” 

Robotech 

“The  more  I knew,  the  less  stress  I’d  be  under.  That  was  the  real  thing,  was  just  to  make 
my  job  easier.” 

Elroy 

“If  it  is  something  I’m  just  curious  about.,  .my  goals  would  differ  from  something  from 
work  that  I have  to  learn.  Usually  those  are  where  you  have  to  get  to  where  you  can 
satisfy  the  business  need  by  this  time.” 

Trogdor 

“I  almost  see  fun  as  a given.” 

“I  feel  very,  very  motivated  to  constantly  be  making  sure  that  I’m  learning  stuff.” 
“So  I also  like  technologies  based  on,  how’s  this  going  to  benefit  my  resume.” 

The  participants’  statements  regarding  their  goals  reveal  many  different  goal  types  operating  at  the 
same  time.  They  mentioned  learning  for  future  utility,  solving  current  problems  in  the  workplace  but  also 
learning  more  than  is  needed  immediately  because  the  knowledge  might  be  useful  later.  In  addition,  they 
intrinsically  enjoyed  working  with  the  new  technology  and  learning  new  things.  Some  also  reported  concerns 
about  others’  opinions  of  their  abilities,  which  could  be  characterized  as  performance  goals.  It  is  important  to 
some  of  them  that  they  are  seen  as  a source  of  information  on  new  technology,  and  that  others  realize  what  they 
have  accomplished. 


Discussion 

Overall,  the  responses  of  the  IT  experts  supported  many  of  the  elements  in  the  proposed  expertise 
model,  even  though  they  worked  in  different  areas  of  IT.  Participants  reported  having  a talent  for  working  with 
technology,  getting  support  in  their  learning  from  others  in  their  physical  or  virtual  community,  experiencing 
flow  while  learning,  using  metacognitive  strategies,  engaging  in  deliberate  practice,  and  holding  multiple  goals. 
Self-taught  IT  workers  had  been  using  technology  from  an  early  age,  linked  new  knowledge  to  prior  knowledge 
when  learning,  and  were  reluctant  to  immediately  ask  questions.  They  found  learning  resources  based  on 
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recommendations  from  friends  or  strangers  in  person  or  online,  weighing  the  advice  they  got  against  their  own 
experience.  While  learning  they  set  up  tasks  for  themselves  and  persisted  in  the  face  of  problems  or  difficulties, 
but  measured  progress  each  in  their  own  way.  The  challenge,  focus,  and  satisfaction  of  the  flow  experience 
kept  them  motivated,  especially  when  they  had  uninterrupted  time.  The  IT  workers  acknowledged  a variety  of 
goals,  including  learning,  performance,  and  future  utility  goals,  either  acting  at  different  times  or 
simultaneously. 

One  limitation  of  the  study  concerns  the  difficulty  of  identifying  experts  in  the  domain,  even  though 
development  of  expertise,  as  opposed  to  a specific  level  of  expertise,  was  the  issue  at  hand.  Although  the 
snowball  method  identified  IT  workers  who  were  at  a level  of  expertise,  they  still  might  not  have  been  at  the 
same  level  of  expertise.  In  addition,  the  IT  experts  studied  may  have  learned  in  a variety  of  settings  in  the  past, 
even  if  they  were  primarily  self-taught.  I eliminated  from  the  study  any  IT  worker  whose  undergraduate  or 
graduate  degree  was  in  computer  science,  computer  engineering,  management  information  systems,  or  a related 
field.  Eliminating  those  who  had  taken  shorter-term  training  courses  would  have  been  impractical  and  probably 
unnecessary,  as  an  IT  worker  with  several  years  of  self-taught  experience  along  with  a few  one-week  training 
courses  is  arguably  more  influenced  by  the  self-teaching  than  the  short  courses.  Therefore  those  whose  formal 
training  was  limited  to  short  courses  were  not  excluded,  but  this  training  experience  may  still  have  affected  the 
participants’  descriptions  of  independent  learning  in  information  technology. 
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Abstract 

Simonson,  Schlosser  and  Hanson  ( 1 999)  argue  that  a new  theory  called  “equivalency  theory  ” is 
needed  to  account  for  the  unique  features  of  the  “teleconferencing”  ( synchronous ) model  ofDE  that  is 
prevalent  in  many  North  American  universities.  Based  on  a comprehensive  meta-analysis  of  the  comparative 
literature  ofDE  (Bernard,  Abrami,  Lou,  Wozney,  Borokhovski,  Wallet,  Wade,  Fiset,  & Huang,  in  press),  we  are 
able  to  assess  empirically  whether  equivalency  has  been  achieved  in  prior  comparative  DE  research.  This 
paper  includes  a brief  summary  of  the  results  of  the  split  between  synchronous  and  asynchronous  patterns  of 
DE,  and  addresses  the  implications  these  data  have  for  developing  separate  theories  of  DE  for  synchronous 
(i.e.,  group-based)  and  asynchronous  (i.e.,  individualized ) applications.  We  examine  data  based  on 
achievement,  attitude  and  retention  outcomes  and  coded  study  features  (i.e.,  methodological,  pedagogical  and 
media)  relating  to  them. 


Introduction 

Over  the  past  several  decades,  two  distinctly  different  patterns  of  distance  education  (DE)  have 
emerged,  along  with  a variety  of  combinations  of  them.  Synchronous  DE  derives  from  early  applications  of 
closed  circuit  TV  on  college  campuses  (e.g..  Carpenter  & Greenhill,  1955;  1958).  In  this  pattern,  two  or  more 
classrooms  in  different  locations  are  joined  in  real  time  and  run,  synchronously,  usually  from  the  originating 
site.  Today,  various  forms  of  audio  and  video  interactive  teleconferencing  technology  are  used  to  unite 
originating  and  remote  classroom  sites.  According  to  Mottet  (1998)  and  Ostendorf  (1997),  this  form  of 
“emulated  traditional  classroom  instruction”  is  the  fastest  growing  form  ofDE  in  U.S.  universities,  and  so  it  is 
important  for  us  to  know  how  it  affects  learners  who  are  involved  in  it. 

Modern  asynchronous  DE  is  a derivative  of  correspondence  education,  where  the  “medium  of 
instruction”  was  the  post  office  and  asynchronicity  was  a result  of  postal  delay.  In  this  pattern,  students  in 
remote  locations  work  independently  or  in  asynchronous  groups,  usually  with  the  support  of  an  instructor  or 
tutor.  Communication  between  the  instructor  and  student  and  among  students  typically  involves  the  Internet, 
although  contact  through  other  media  is  (e.g.,  telephone)  is  often  used.  Asynchronous  DE,  then,  fits  within 
Keegan’s  (1996)  definitional  criteria  of  teaching  and  learning  “anyplace,  anytime,”  whereas  synchronous  DE  is 
both  time  and  place  dependent. 
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Simonson,  Schlosser  and  Hanson  ( 1999)  argue  that  separate  theories  of  DE  are  needed  to  account  for 
the  different  underlying  premises  of  these  two  models.  What  we  have  referred  to  as  asynchronous  DE  springs 
from  theoretical  perspectives  that  emphasize  “independence  and  autonomy  of  the  learner,  industrialization  of 
teaching,  and  interaction  and  communication.  These  classical  theories  emphasize  the  notion  that  distance 
education  is  a fundamentally  different  form  of  education’’  (p.  74).  By  contrast,  Simonson  et  al.  argue  that  a 
theory  of  synchronous  DE  should  focus  on  providing  individual  but  equivalent  experiences  for  the  classroom 
and  DE  learners.  They  say  that  “Recent  emerging  theories  based  on  the  capabilities  of  new  interactive 
telecommunication-based  audio  and  video  systems  suggest  that  distance  education  may  not  be  a distinct  field  of 
education”  (p.  74). 

We  can  take  no  position  on  the  theoretical  arguments  contained  in  the  above  quotations,  except  to  say 
that  there  is  one  important  distinction  that  is  evident  in  the  two  patterns.  Synchronous  DE,  by  definition,  is 
inextricably  tied  to  classroom  instruction.  In  this  sense,  it  may  be  viewed  as  an  extension  of,  or  as  a special  case 
of  classroom  instruction.  Asynchronous  DE,  by  contrast,  is  not  necessarily  bound  by  conditions  that  exist  in  a 
classroom.  This  fact  suggests  that  the  two  patterns  may  encompass  different,  but  related,  sets  of  teaching  and 
learning  skills. 

An  issue  that  arises  is  upon  what  grounds  should  “equivalence”  be  judged.  Simonson  et  al.  (1999)  offer 
this  suggestion:  “...  the  outcomes  of  a learning  experience  are  those  obvious,  measurable,  and  significant 
changes  that  occur  cognitively  and  affectively  in  learners  because  of  their  participation  in  the  course  or  unit”  (p. 
72).  They  distinguish  between  “instructor-determined  outcomes”  and  “learner-determined  outcomes.”  The 
former  are  presumably  the  traditional  measures  of  achievement  and  course  satisfaction  (e.g.,  attitude  measures), 
while  the  latter  are  delayed  measures  of  skills  application  and  follo\wup  enrollment  in  similar  courses.  We 
argue  that  another  related  measure  of  learner-determination  is  the  retention  rate  (or  conversely,  the  dropout  rate) 
in  a course. 

The  research  literature  of  DE  contains  many  studies  of  both  synchronous  and  asynchronous  DE  as  they 
compare  to  traditional  classroom  instruction.  In  2001,  we  undertook  to  review,  quantitatively  (meta-analysis), 
these  comparative  studies  dating  from  1985  through  2002.  These  were  some  of  the  findings:  1)  there  is  wide 
variability  in  achievement,  attitude  and  retention  outcomes  (i.e.,  some  classrooms  vastly  outperform  DE  and 
vice  versa);  2)  study  features  describing  differences  in  research  methodology  account  for  a substantial 
proportion  of  the  variance  in  outcomes;  3)  overall,  elements  of  pedagogy  account  for  more  variation  than 
elements  of  media  usage  (Clark,  1983,  1994);  and  4)  a number  of  study  features  (both  pedagogy  and  media)  are 
predictors  of  the  differences  between  DE  and  classroom  instruction  (Bernard,  Abrami,  Lou,  Wozney, 
Borokhovski,  Wallet,  Wade,  Fiset,  & Huang,  in  press).  Here  we  will  present  a subset  of  these  results  relating  to 
synchronous  and  asynchronous  DE  to  describe  how  these  patterns  differ  in  terms  of  achievement,  attitude  and 
retention  outcomes.  We  will  focus  especially  on  the  comparative  effects  of  synchronous  DE  and  consider 
whether  the  DE  part  of  this  dyad  differs  from  its  classroom  counterpart.  We  will  also  examine  the  effects  of 
individual  pedagogy  and  media  study  features  in  an  attempt  to  suggest  what  might  be  changed  in  synchronous 
DE,  in  particular  to  increase  its  “equivalence”  to  classroom  instruction. 

The  Nature  of  Meta- Analysis 

An  axiom  of  modern  cognitive  science  is  that  knowledge  is  situated.  Likewise,  the  evidence  from  a 
primary  investigation  should  also  be  situated  in  the  larger  context  of  other  investigations  so  that  evidence  is 
organized,  accumulated  and  otherwise  synthesized — a cornerstone  of  any  science.  Meta-analysis,  or  quantitative 
research  synthesis  (Glass,  McGaw  & Smith,  1981;  Hedges  & Olkin,  1985),  is  the  most  unbiased  way  of 
conducting  a systematic  review  and  estimating  the  treatment  effects  associated  with  a collection  of  primary 
studies  addressing  different  facets  of  a common  question.  The  methodology  also  provides  ways  of  asking  and 
answering  more  detailed  questions  concerning  the  influence  of  coded  moderator  variables.  Arguably  the 
greatest  contribution  of  meta-analysis  is  the  introduction  of  the  concepts  of  precision,  systematicity  and 
replicability  to  the  conduct  of  research  reviews  (Abrami,  Cohen,  & d’ Apollonia,  1988;  Bernard  & Naidu,  1990). 

Method 


Definition  of  DE 

Our  definition  of  DE  derives  from  Keegan’s  ( 1996)  basic  set  of  characteristics  that  distinguish  DE 
from  other  forms  of  instruction.  This  definition  allowed  for  the  inclusion  of  both  synchronous  and  asynchronous 
patterns  of  DE. 
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• The  semi  -permanent  separation  (place  and/or  time)  of  learner  and  instructor  during  planned  learning  events. 

• The  presence  of  planning  and  preparation  of  the  learning  materials,  student  support  services,  and  the  final 

recognition  of  course  completion  by  an  educational  organization. 

• The  provision  of  two-way  media  to  facilitate  dialogue  and  interaction  between  students  and  the  instructor 

and  among  students. 

Inclusion/Exclusion  Criteria 


To  be  included  in  this  meta-analysis,  each  study  had  to  meet  the  following  basic  inclusion/exclusion 

criteria: 

1.  Involve  an  empirical  comparison  of  DE  as  defined  in  this  meta-analysis.  Studies  comparing  DE  with 
national  standards  or  norms,  rather  than  a control  condition,  were  excluded. 

2.  Involve  “distance  from  instructor”  as  a primary  condition  of  the  DE  condition.  DE  with  some  face-to-face 
meetings  (fewer  than  50%)  was  included. 

3.  Report  measured  outcomes  for  both  experimental  and  control  groups. 

4.  Report  sufficient  statistical  data.  Studies  with  insufficient  data  for  effect  size  calculations  (e.g.,  with  means 
but  no  standard  deviations  or  no  inferential  statistics  or  no  sample  size)  were  excluded. 

5.  Be  publicly  available  or  archived. 

6.  Include  at  least  one  achievement,  attitude  or  retention  outcome  measure. 

7.  Include  an  identifiable  level  of  learner.  All  levels  of  learners  from  kindergarten  to  adults,  whether 
informally  schooled  or  professionally  trained,  were  admissible. 

8.  Be  published  or  presented  no  earlier  than  1985  and  no  later  than  December  of  2002. 

9.  Include  outcome  measures  that  were  the  same  or  comparable.  If  the  study  explicitly  stated  that  different 
exams  were  used  for  the  experimental  and  control  groups,  the  study  was  excluded. 

Data  Sources  and  Search  Strategies 

The  studies  used  in  this  meta-analysis  were  located  through  a comprehensive  search  of  publicly 
available  literature  from  1985  through  December  of  2002.  Electronic  searches  were  performed  on  the  following 
databases:  ABI/Inform , Compendex,  Cambridge  Scientific  Abstracts,  Canadian  Research  Index,  Communication 
Abstracts,  Digitcd  Dissertations  on  ProQuest,  Dissertation  Abstracts,  Education  Abstracts,  ERIC,  Psyclnfo,  and 
Social  SciSearch.  Web  searches  were  performed  using  Google,  AlltheWeb,  and  Teoma  search  engines.  A 
manual  search  was  performed  in  ComAbstracts,  Educational  Technology  Abstracts;  in  several  distance  learning 
journals,  including  The  American  Journal  of  Distance  Education,  Distance  Education,  Journal  of  Distance 
Education,  Open  Learning,  and  Journal  of  Telemedicine  and  Telecare;  and  in  several  conference  proceedings, 
including  AECT,  AACE,  AERA,  CADE,  EdMedia,  E-Learn,  SITE,  and  WebNet.  In  addition,  the  reference  lists  of 
several  earlier  reviews  were  examined  for  possible  inclusions.  Although  search  strategies  varied  depending  on 
the  tool  used,  generally,  search  terms  included  “distance  education,”  “distance  learning,”  “open  learning”  or 
“virtual  university,”  AND  (“traditional,”  “lecture,”  “face-to-face”  or  “comparison”). 

Outcomes  of  the  Searches 

In  total,  more  than  5,000  research  abstracts  concerning  DE  and  traditional  classroom-based  instruction 
were  examined  and  862  full-text  potential  includes  retrieved.  Each  of  the  studies  retrieved  was  read  by  two 
researchers  for  possible  inclusion  using  the  inclusion/exclusion  criteria.  The  initial  inter-rater  agreement  as  to 
inclusion  was  89%.  Any  study  that  was  considered  for  exclusion  by  one  researcher  was  crosschecked  by 
another  researcher.  Two  hundred  and  thirty-two  (232)  studies  met  all  inclusion  criteria  and  were  included  in  this 
meta-analysis;  630  were  excluded. 


Extraction  of  Effect  Sizes 

Effect  sizes  were  extracted  from  each  study.  Where  possible,  descriptive  data  (i.e.,  means  and  standard 
deviations)  were  used  to  calculate  Cohen’s  d (see  Equation  1). 


d = 

i 


Y -Y 

Experimental  Control 


$ Pooled 


(i) 
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Where  these  data  were  not  available,  effect  sizes  were  calculated  or  estimated  from  statistical  test  data 
(e.g.,  /-ratios,  probabilities)  based  on  equations  provided  by  Glass,  McGaw,  and  Smith  (1981)  and  Hedges, 
Shymansky,  and  Woodworth  (1989).  Cohen’s  d was  converted  to  Hedges’  g (Hedges  & Olkin,  1985)  to  correct 
for  the  influence  of  sample  size.  Hedges’  g+  (i.e.,  average  effect  size)  was  then  calculated  for  each  relevant 
subgroup  and  for  each  of  the  three  measures  along  with  homogeneity  of  effect  size  statistics. 

Results 

Meta-analysis  is  a descriptive  review  technique  and  as  such  provides  a characterization  of  a large  body 
of  quantitative  evidence.  While  effect  sizes  are  tested  for  significance  (g+  = 0),  this  does  not  involve  the  same 
sort  of  inferential  testing  that  is  often  done  in  original  research.  One  way  of  thinking  of  a comprehensive  meta- 
analysis is  as  an  approximation  of  differences  in  the  population  that  far  surpasses  the  evidence  provided  by  any 
single  study. 

Statistical  Analysis 

In  total,  232  studies  yielding  688  independent  effect  sizes  (i.e.,  outcomes)  were  analyzed.  This  was 
based  on  totals  of  57,019  students  ( k = 321)  with  achievement  outcomes,  35,365  students  ( k = 262)  with  attitude 
outcomes,  and  ( N = 57,916,029)  students  ( k = 105)  with  retention  outcomes.  The  N reported  here  for  retention 
was  reduced  proportionally  to  3,744,869  to  avoid  overestimation  based  on  a longitudinal  California  study  of 
retention  in  junior  colleges.  The  number  of  outcomes  was  further  reduced  when  they  were  classified  as  either 
synchronous  or  asynchronous  patterns. 

The  statistical  findings  that  are  relevant  to  the  question  being  examined  in  this  paper  are  shown  in  Table 
1.  In  this  table,  zero  defines  no  mean  difference  between  the  DE  condition  and  the  classroom  condition.  Positive 
mean  effect  sizes  favor  DE  over  the  classroom.  Negative  effect  sizes  indicate  the  reverse.  Bear  in  mind  as  you 
examine  these  mean  effect  sizes  that  wide  variability  surrounds  each  of  them  (i.e.,  homogeneity  of  effect  size 
was  violated).  Hedges  and  Olkin  (1985)  warn  against  a strong  interpretations  of  mean  effect  sizes  when 
assumptions  of  homogeneity  of  effect  size  are  violated.  To  dramatize  this  point,  we  found  effect  sizes  for 
synchronous  achievement  outcomes  to  vary  from  -1.14  to  0.97.  For  asynchronous  outcomes,  the  minimum  g was 
-1.31  and  the  maximum  g was  1.31.  That  says  that  some  applications  of  DE,  both  synchronous  and 
asynchronous,  far  outperformed  their  classroom  comparison  group  and  some  far  underperformed  it.  However, 
the  pattern  of  results  is  interesting  and  bears  consideration. 

First,  all  but  two  average  effects  sizes  are  significantly  greater  than  zero.  As  with  all  statistical  tests, 
higher  degrees  of  freedom  provide  more  power  and  a more  sensitive  test,  so  that  at  least  one  test  (synchronous, 
retention  outcomes)  may  represent  an  artifact  of  this. 

There  is  evidence  for  synchronous  DE,  based  on  an  analysis  of  achievement  outcomes,  that  students  in 
the  DE  condition  are  not  performing  as  well  as  their  classroom  companions,  on  average.  This  effect  in  favor  of 
the  classroom  condition  appears  to  be  even  more  dramatic  for  attitude  outcomes.  By  contrast,  DE  student 
experiencing  asynchronous  DE  outperformed  their  classroom  equivalents  on  achievement  measures  and 
performed  equally  well  in  terms  of  attitude  outcomes  (although  the  effect  size  is  negative). 


Table  1.  Weighted  Mean  Effect  Sizes  for  Achievement,  Attitude  and  Retention  Outcomes  (Synchronous  and 
Asynchronous) 


Achievement 

Attitude 

Retention 

Categories  of  DE 

8+ 

SE 

g+ 

SE 

S+ 

SE 

Synchronous  DE 

-0.102* 

(*=92) 

0.024 

-0.185* 
(k  = 83) 

0.022 

0.005 
(*  = 17) 

0.034 

Asynchronous  DE 

0.053* 
(Ic  = 17 4) 

0.012 

-0.003 

(*=71) 

0.019 

-0.093* 
(k  = 53) 

0.021 

*p  < .05  Note:  All  mean  effect  sizes  are  heterogeneous. 
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The  situation  for  retention  outcomes  is  essentially  the  reverse  of  the  above.  The  retention  rate  (i.e., 
opposite  of  dropout)  for  synchronous  comparisons  was  zero,  while  significantly  more  students  dropped  out  of 
asynchronous  DE  than  their  classroom  equivalents.  This  problem  with  DE  has  existed  from  the  early  days  of 
correspondence  education  (e.g.,  Bernard  & Amundsen,  1988). 

This  analysis  does  not  suggest  that  DE  and  classroom  conditions  in  synchronous  DE  cannot  be 
equivalent,  just  that  when  they  have  been  comp  ared  in  a large  number  of  studies,  they  are  not.  Beyond  the 
intuitive  notion  that  remote-site  students,  being  on  the  receiving  end  of  a technologically  mediated  classroom 
(virtual  classroom),  may  not  feel  as  included,  may  not  get  to  participate  as  much,  find  it  harder  to  concentrate, 
and/or  may  not  get  the  timely  feedback  that  they  need,  we  have  little  evidence  to  explain  this  apparent  disparity. 
A scarcity  of  information  in  the  literature,  especially  of  the  conditions  in  the  classroom,  made  it  difficult  for  us 
to  find  sets  of  robust  predictors  that  would  have  helped  us  develop  a more  complete  picture  or  empirical  model 
of  the  instructional  characteristics  that  make  a difference.  See  Bernard  et  al.,  (in  press)  and  for  the  complete 
analysis  and  Bernard,  Abrami,  Lou,  and  Borokhovski  (in  press)  for  an  extended  discussion  of  the 
methodological  state  of  DE  research. 

Study  Feature  Predictors  of  Outcomes 

In  an  attempt  to  further  explain  the  results  just  presented,  we  used  weighted  multiple  regression 

1/^2 

(WMR,  where  the  weighting  factor  is  the  inverse  of  the  population  variance,  1/0  , approximated  in  an 

equation  from  Hedges  & Olkin,  1985,  p.  174)  using  individually  coded  study  features  as  predictors  andg  as  the 
outcome  variable.  The  study  features  were  categorized  as:  1)  13  methodological  predictors;  2)  9 pedagogical 
predictors;  and  3)  9 media  use  predictors,  and  were  entered  into  blocks  in  WMR.  Methodology  was  entered 
first,  followed  by  pedagogy.  In  a second  analysis,  methodology  again  was  entered  first,  followed  by  media  use. 
In  this  way  we  were  able  to  assess  the  effects  of  pedagogy  and  media  use,  independently,  after  variation  due  to 
methodological  differences  was  removed.  We  did  this  for  synchronous  and  asynchronous  DE  outcomes 
separately.  In  presenting  these  findings,  we  must  warn  against  the  over-interpretation  of  individual  study 
features.  While  all  of  the  ones  that  are  described  here  were  significant,  a certain  degree  of  collinearity  among 
them  makes  drawing  strong  conclusions  problematic. 

Synchronous  DE.  The  question  that  we  will  be  attempting  to  answer  here  is:  If  synchronous  DE  and  its 
classroom  counterparts  are  not  equivalent,  how  can  we  make  them  more  so?  Since  from  our  overall  analysis,  we 
have  established  that  in  synchronous  DE,  the  classroom  condition  had  better  achievement  and  better  attitudes 
towards  instruction  than  the  DE  condition,  what  can  be  added  to  or  changed  about  the  DE  condition  to 
ameliorate  the  situation?  In  looking  at  Table  2,  which  displays  the  significant  predictors  of  achievement  and 
attitudes  (an  analysis  of  retention  outcomes  produced  no  significant  predictors),  we  see  study  features  that  favor 
the  classroom  on  the  left  and  those  that  favor  DE  on  the  right  (this  was  determined  by  the  sign  of  the  regression 
weight  associated  with  each  predictor). 

The  most  striking  feature  contained  in  the  set  of  predictors  for  synchronous  DE  is  the  large  role  that 
personal  contact  with  the  instructor  and  other  students,  either  face-to-face  or  mediated  by  technology,  plays  in 
achievement  and  attitude  outcomes.  This  agrees  with  much  of  the  previous  literature  of  DE,  which  cites  feelings 
of  isolation  as  a significant  contributor  to  problems  such  as  dropout.  Two  other  predictors  were  significant, 

“Use  of  one-way  TV-video”  and  “Use  of  systematic  instructional  design.”  In  this  case,  TV -video  does  not  refer 
to  the  means  of  communication  (i.e.,  videoconferencing).  It  is,  instead,  the  use  of  televised  materials  and 
content  delivered  at  a distance.  It  is  conceivable  that  such  materials  offer  a welcome  alternative  to  lecture  and 
other  forms  of  teacher-dominated  instruction.  It  is  not  surprising  to  find  that  explicit  prior  planning  improves 
the  quality  of  DE.  Had  there  been  more  information  about  classroom  conditions,  we  might  have  found  the  same 
thing  for  face-to-face  instruction. 

Table  2.  Summary  of  Study  Features  that  Significantly  Predict  Outcomes  in  Synchronous  DE 

Synchronous  DE 

Favor  Classroom  Instruction  (-) Favor  Distance  Education  (+) 


Achievement 


Achievement 


• Face-to-face  meetings  with  the  • Face-to-face  contact  with  other  students 

instructor  • use  of  one-way  TV-video 

• Use  of  the  telephone  to  contact 
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instructor 


Attitudes 

Attitudes 

• Opportunity  for  face-to-face  contact  with 

• Use  of  systematic  ID 

other  students 

• Opportunity  for  mediated  communication 

• Use  of  one-way  TV-video 

with  the  instructor 

• Instructor/student  contact  encouraged 

• Use  of  the  telephone  to  contact  instructor 

Asynchronous  DE.  Table  3 is  structured  in  the  same  way  as  the  previous  table.  Here  again, 
communication  and  the  use  of  mediated  content  delivery  play  important  roles.  However,  in  contrast  to 
synchronous  DE,  an  explicit  learning  strategy,  “Problem-based  learning”  emerged  as  an  important  predictor  of 
both  achievement  and  attitude  outcomes.  This  is  particularly  interesting  when  combined  with  the  appearance  of 
computer-mediated  communication.  It  is  possible  that  this  pairing  suggests  a positive  effect  for  computer-based 
collaborative  learning. 

Table  3 . Summary  of  Study  Features  that  Significantly  Predict  Outcomes  in  Asynchronous  DE 

Asynchronous  DE 

Favor  Classroom  Instruction  (-) Favor  Distance  Education  (+) 


Achievement 

Achievement 

No  significant  predictors 

• Use  of  problem-based  learning  strategies 

• Opportunity  for  mediated 

communication  with  the  instructor 

• Advance  information  given  to  students 

• Use  of  one-way  TV-video 

Attitudes 

Attitudes 

• Use  of  the  Web 

• Use  of  problem-based  learning  strategies 

• Use  of  computer-mediated 
communication 

• Use  of  computer-based  instruction 

Discussion 

One  of  the  things  that  meta-analysts  quickly  come  to  realize  is  that  they  are  “prisoners”  of  the  data  and 
the  previous  efforts  of  primary  researchers.  Findings  are  based,  by  necessity,  on  what  can  be  gleaned  fromthe 
literature;  nothing  new  can  be  added.  In  this  study,  lack  of  study  information,  particularly  regarding  to  the 
unreported  characteristics  of  the  classroom  condition,  frustrated  our  efforts  to  delve  more  deeply  into  the  nature 
of  instructional  study  features  and  their  relationship  to  the  overall  findings.  Why  do  effect  sizes  vary  so  widely, 
even  after  they  are  categorized  as  synchronous  or  asynchronous?  What  should  be  included  in  synchronous  DE 
to  make  it  as  effective  as  (equivalent  to)  classroom  instruction,  or  at  least  the  best  it  can  be?  What  practices 
should  be  avoided?  What  are  the  ranges  of  instructional  practices  and  learning  strategies  that  best  support 
achievement  and  satisfaction  in  asynchronous  DE?  Answers  to  these  kinds  of  questions  did  not  emerge  as 
clearly  as  they  could  have  if  the  literature  were  more  complete. 

Nevertheless,  some  interesting  findings  did  emerge  concerning  the  nature  of  synchronous  and 
asynchronous  DE.  We  found  that  synchronous  DE,  in  particular,  produced  average  effect  sizes  that  favored  the 
classroom  condition  for  both  achievement  and  attitude  outcomes.  We  also  found  wide  variation  around  these 
means.  This  suggests  that  synchronized  DE  classrooms  can  produce  outcomes  that  are  at  least  equivalent  to 
“live”  classrooms  but  that  this  doesn’t  always  happen  (in  fact,  in  a majority  of  cases  it  doesn’t).  The  reasons  for 
this  may  be  manifold,  ranging  from  the  application  of  teacher-centered  instructional  techniques,  which  might 
not  be  as  engaging  in  a mediated  form,  to  failures  associated  with  the  technologies  involved  (either  through 
poor  application  or  failures  of  the  technology  itself).  It  is  very  likely  that  a new  set  of  instructional  skills, 
beyond  those  applied  in  the  classroom,  is  required  for  instructors  to  meet  the  challenges  of  synchronous  DE. 
While  the  tested  principles  of  pedagogy  (e.g.,  motivation,  engagement,  interactivity,  evaluation,  feedback)  may 
generally  apply,  their  application  in  synchronous  mediated  DE  environments  may  require  experience  and 
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possibly  even  special  training.  It  is  likely  that  many  of  the  students  involved  in  these  studies  had  little  if  any 
prior  experience  with  DE,  and  consequently  did  not  know  what  to  expect  from  DE  or  how  to  engage  effectively 
in  this  new  form  of  educational  experience.  If  that  is  the  case,  this  situation  may  change  radically  as  more 
learning  opportunities  are  offered  and  more  students  partake  of  them. 

One  of  the  prescriptions  that  did  emerge  regarding  synchronous  DE,  in  particular,  is  that  it  should 
involve  more  direct  personalized  contact  between  students  and  the  instructor  and  among  students.  This  is 
tantamount  to  saying:  “make  synchronous  DE  more  like  face-to-face  instruction,”  and  it  is  arguable  that  this  is 
not  a bad  idea  if  in  fact  synchronous  DE  is  to  be  regarded  as  a special  case  of  face-to-face  instruction.  If  this  is 
not  feasible,  as  it  often  is  not  in  DE,  serious  efforts  should  be  made  to  compensate  for  this  deficit  through 
various  mediated  options  (Simonson,  Schlosser  & Hanson  make  this  very  point). 

We  also  found  some  interesting  relationships  regarding  asynchronous  DE.  In  another  paper  (Bernard  et 
ah,  in  press)  we  provide  evidence  of  the  precedence  that  pedagogy  takes  over  media  in  DE,  especially  in 
asynchronous  settings,  a point  that  Richard  Clark  ( 1983,  1994)  has  made  for  years  regarding  all  forms  of 
technology  applied  to  instruction.  While  it  is  axiomatic  that  DE  requires  technology,  it  must  function  in  the 
service  of  something — content  delivery  (e.g.,  TV/video,  Web  resources),  communication  between  student  and 
instructor  (e.g.,  clarification  and  feedback),  and/or  communication  among  students  (e.g.,  mediated 
communication) — that  leads  to  learning  success.  We  have  found  direct  evidence  here  of  all  of  these  elements. 
We  have  found  indirect  evidence  that  collaborative  learning  (i.e.,  the  combination  of  mediated  communication 
and  problem-based  learning)  is  present.  Direct  evidence  of  the  effectiveness  of  collaborative  learning  strategies 
may  emerge  as  more  studies  of  this  relatively  recent  approach  to  learning  are  conducted.  It  is  also  possible  that 
advances  in  multi-media,  CBI,  simulation  media  etc.  may  herald  the  new  era  of  interactiveness  posited  by 
Ullmer  (1984)  and  Cobb  (1997). 

This  empirical  assessment  of  the  comparative  state  of  affairs  in  studies  of  synchronous  and 
asynchronous  DE  begs  the  question  of  whether  a new  theory — “equivalency  theory”  or  any  other,  for  that 
matter — is  required  to  understand  and  to  make  progress  in  the  development  of  DE.  Meta-analysis  cannot  answer 
such  questions,  even  when  the  question  involves  a testable  hypothesis.  This  meta-analysis  has  demonstrated, 
however,  that  we  have  a long  way  to  go  before  we  can  say  with  certainty  that  either  form  of  DE  will  reliably 
offer  educational  opportunities  that  equal  or  exceed  that  which  is  currently  called  “traditional  classroom 
instruction.” 


References 

Abrami,  P.  C.,  Cohen,  P.,  & d’Apollonia,  S.  (1988).  Implementation  problems  in  meta-analysis.  Review  of 
Educational  Research,  58,  151-179. 

Bernard,  R.  M.,  Abrami,  P.  C.,  Lou,  Y.  & Borokhovski,  E.  (in  press).  A methodological  morass?  How  we  can 
improve  the  quality  of  quantitative  research  in  distance  education.  Distance  Education. 

Bernard,  R.  M.,  Abrami,  P.  C.,  Lou,  Y.,  Wozney,  L.,  Borokhovski,  E.,  Wallet,  P.  A.,  Wade,  A.,  Fiset,  M.,  & 

Huang,  B.  (in  press).  How  does  distance  education  compare  to  classroom  instruction?  A meta-Analysis 
of  the  empirical  literature.  Review  of  Educational  Research. 

Bernard,  R.  M.,  & Naidu,  S.  (1990).  Integrating  research  into  practice:  The  use  and  abuse  of  meta-analysis. 
Canadian  Journal  of  Educational  Communication,  19,  171-195. 

Bernard,  R.  M.,  & Amundsen,  C.  L.  (1989).  Antecedents  to  dropout  in  distance  education:  Does  one  model  fit 
all?  Journal  of  Distance  Education,  4(2),  25-46. 

Carpenter,  C.  R.,  & Greenhill,  L.  P.  (1955  & 1958).  An  investigation  of  closed-circuit  television  for  teaching 
university  courses  (Reports  1 & 2).  University  Park:  Pennsylvania  State  University. 

Clark,  R.  E.  (1983).  Reconsidering  research  on  learning  from  media.  Review  of  Educational  Research,  53,  445- 
459. 

Clark,  R.  E.  (1994).  Media  will  never  influence  learning.  Educational  Technology  Research  and  Development, 
42(2),  21-29. 

Cobb,  T.  (1997).  Cognitive  efficiency:  Toward  a revised  theory  of  media.  Educational  Technology  Research 
and  Development,  45(4),  21-35. 

Glass,  G.  V.,  McGaw,  B.,  & Smith,  M.  L.  (1981).  Meta-analysis  in  social  research.  Beverly  Hills,  CA:  Sage. 

Hedges,  L.  V.,  & Olkin,  I.  (1985).  Statistical  methods  for  meta-analysis.  Orlando,  FL:  Academic  Press. 

Hedges,  L.  V.,  Shymansky,  J.  A.,  & Woodworth,  G.  (1989).  A practical  guide  to  modern  methods  of  meta- 
analysis. (ERIC  Document  Reproduction  Service  No.  ED  309  952) 

Keegan,  D.  (1996).  Foundations  of  distance  education.  (3rd  Ed).  London:  Routledge. 


108 


Mottet,  T.  P.  (1998).  Interactive  television  instructors  = perceptions  of  students  = nonverbal  responsiveness  and 
effects  on  distance  teaching.  Dissertation  Abstracts  International,  59(02),  460A.  (University 
Microfilms  No.  AAT98-24007) 

Ostendorf,  V.  A.  (1997).  Teaching  by  television  [Theme  issue:  Teaching  and  learning  at  a distance:  What  it 
takes  to  effectively  design,  deliver,  and  evaluate  programs].  New  Directions  for  Teaching  and 
Learning,  no.l,  51-58. 

Sedlmeier,  P.,  & Gigerenzer,  G.  (1989).  Do  studies  of  statistical  power  have  an  effect  on  the  power  of  studies? 
Psychological  Bulletin,  105,  309-316. 

Simonson,  M.,  Schlosser,  C.,  & Hanson,  D.  (1999).  Theory  and  distance  education:  A new  discussion. 

American  Journal  of  Distance  Education;  (13)  1,60-75. 

Ullmer,  E.  J.  (1994).  Media  and  learning:  Are  there  two  kinds  of  truth?  Educational  Technology  Research  & 
Development,  42(1),  21-32. 


109 


Effects  of  Long-term  Professional  Development  Training  in  Technology 
Integration  on  Teacher  and  Student  Performance 

Heidi  C.  Blair 
Arizona  State  University 

Howard  Sullivan 
Arizona  State  University 

Introduction 

The  Arizona  Classrooms  of  Tomorrow  Today  (AZCOTT)  program  was  a twelve- month  graduate  level 
professional  development  program  designed  to  aid  Grade  3-8  teachers  with  infusing  technology  into  their 
instruction  and  to  support  student  achievement.  Originally  a Preparing  Tomorrow’s  Teachers  to  Use 
Technology  (PT3)  grant  from  the  U.S.  Department  of  Education  funded  the  AZCOTT  program,  a partnership 
between  five  suburban  school  districts  and  one  university  in  the  southwestern  United  States.  An  Arizona  Board 
of  Regents  Improving  Teacher  Quality  Grant  provided  funding  for  the  evaluation  year  of  2003-2004. 

The  AZCOTT  Advisory  Board  requested  an  evaluation  of  the  effectiveness  of  the  program  to  enable 
them  to  make  data-driven  decisions  regarding  future  funding  and  expansion  of  the  program.  Over  the  last  few 
years,  billions  of  dollars  have  been  spent  on  educational  technology  and  related  professional  development 
across  the  nation  (“Technology  Counts”,  2004),  it  is  then  reasonable  to  eximine  the  return  on  that  investment. 
This  study  was  designed  to  examine  the  technology  skills  and  use  of  AZCOTT  teachers  and  their  students  and  a 
comparison  group  of  Non-AZCOTT  teachers  and  students.  Many  professional  development  program 
evaluations  analyze  the  types  of  activities  teachers  plan  (Becker  and  Ravitz,  2001)  and  teacher  attitudes 
(Christensen,  2002).  In  addition  to  these  two  data  types,  this  evaluation  also  examined  student  performance 
data. 

The  four  primary  questions  addressed  by  the  evaluation  were:  1)  Does  the  AZCOTT  program  influence 
the  frequency  of  digital  technology  activities  that  teachers  use  with  their  students  and  that  students  perform  in 
the  classroom?  2)  Does  the  program  influence  student  performance  of  computer  skills?  3)  Does  the  program 
influence  student  self-reports  of  their  technology  skills?  and  4)  Does  the  program  influence  student  self-reports 
of  their  use  of  technology  for  classroom  activities?  Also,  investigated  were  the  students’  ability  to  select 
appropriate  software  tools  for  given  tasks  and  teacher  attitudes  toward  the  AZCOTT  program. 

Description  of  the  AZCOTT  Program 

The  Arizona  Classrooms  of  Tomorrow  Today  professional  development  program  had  two  main 
components:  a long-term  professional  development  program  and  increased  access  to  technology  for  teachers 
and  students.  A team  of  teachers  from  each  of  the  five  partner  districts,  17  teachers  total,  participated  in  60 
hours  of  professional  development  over  13  months.  In  its  publication  of  standards  for  professional  development, 
the  National  Staff  Development  Council  (2001)  asserts  that  long-term/sustained  professional  development 
programs  are  necessary  for  effective  learning  to  take  place.  The  focus  of  the  program  was  on  the  acquisition  and 
integration  of  technology  integration  strategies  into  standards -based  instructional  units  and  on  increasing 
technology  skills.  Each  school  district  increased  the  access  AZCOTT  teachers  and  their  students  had  to 
computers  by  purchasing  new  equipment,  redistributing  existing  equipment,  or  providing  increased  computer 
lab  access. 

The  professional  development  sessions  were  conducted  on  13  Saturdays  between  May  2003  and  May 
2004.  All  sessions  were  held  in  a computer  lab  at  the  partner  university.  The  curriculum  addressed  the  National 
Educational  Technology  Standards  for  Teachers  (NETS-T,  2000)  and  the  Arizona  Technology  Education 
Standards  (Arizona  Department  of  Education,  2000).  The  curriculum  focused  on  developing  standards -based 
instructional  units  that  integrated  technology  into  both  teaching  and  learning  activities,  supporting  learners 
through  the  use  of  graphic  organizer  creation  software  (e.g.  Inspiration®),  identifying  Internet  resources,  and 
implementing  technology  integration  strategies.  The  60  hours  consisted  of  45  classroom  hours  from  Technology 
Integration  in  the  Classroom  taught  by  the  partner  university  and  an  additional  15  classroom  hours  addressing 
project-based  learning  and  the  use  of  video  to  document  classroom  practice. 
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Support  materials  and  resources  included  $80  per  AZCOTT  teacher  to  purchase  instructional  materials 
to  support  the  implementation  of  the  program.  A yearlong  subscription  to  the  online  resource  site  TaskStreamP 
was  provided  for  creating  and  publishing  instructional  units.  Each  teacher  received  a copy  of  Teaching  with 
Technology:  Creating  Student -Centered  Classrooms  and  of  National  Educational  Technology  Standards  for 
Students:  Connecting  Curriculum  and  Technology.  To  expand  the  AZCOTT  teachers’  exposure  to  technology 
integration  strategies,  the  registration  fees  for  two  state  educational  technology  conferences  were  also  paid. 

The  increased  access  to  technology  component  of  the  AZCOTT  program  was  deemed  essential  since 
teachers  often  state  that  if  they  had  more  access  to  technology,  they  would  integrate  it  better  into  their 
instruction  (Kopcha,  2004).  Strategies  employed  by  AZCOTT  partner  districts  for  increasing  teacher  and 
student  access  to  computers  included:  1)  ten  to  20  laptop  computers  were  placed  on  a mobile  cart  with  wireless 
connectivity  to  the  Internet;  2)  the  number  of  desktop  computers  in  a classroom  was  increased  by  five,  and  3) 
the  amount  of  computer  lab  time  allotted  to  AZCOTT  teachers  was  increased.  Some  districts  and  schools 
provided  additional  peripheral  equipment  including  LCD  projectors,  digital  cameras,  digital  video  cameras,  and 
printers.  The  sustained  professional  development  training  design  is  critical  if  the  increased  student  access  to 
computers  is  to  reach  its  full  potential  in  the  classroom  (Sandholtz,  Ringstaff,  & Dwyer,  1997). 

Method 

Participants 

Participants  were  32  Grade  3-8  teachers  and  approximately  800  students  from  five  school  districts  in 
the  southwestern  United  States.  Four  of  the  five  districts  had  different  teachers  participate  in  the  AZCOTT 
program  during  the  2002-2003  school  year.  Free  or  reduced  lunch  percentages  ranged  from  38%  to  89%  and 
minority  populations  ranged  from  20%  to  80%.  The  participants  were  divided  into  two  groups,  those 
participating  in  the  AZCOTT  program  (AZCOTT  teachers  and  students)  and  a comparison  group  (Non- 
AZCOTT  teachers  and  students).  The  comparison  group  was  selected  prior  to  the  study  from  within  the  same 
five  partner  school  districts  to  represent  the  same  grade  levels  and  similar  demographic  characteristics.  Data 
collected  at  the  beginning  of  the  study  indicated  very  similar  beginning-of-study  technology  skills  and  use 
between  the  AZCOTT  and  Non-AZCOTT  teachers. 

Each  of  the  two  teacher  groups  included  three  teachers  from  3rd  grade,  five  from  4th  grade,  four  from 
6th  grade,  three  from  7th  grade,  and  two  from  8th  grade.  The  AZCOTT  teacher  teams  volunteered  to  participate  in 
the  program  as  a team.  The  Non-AZCOTT  teachers  were  asked  to  participate  in  the  evaluation  by  their  site 
administrator. 

Materials 

AZCOTT  teachers  and  Non-AZCOTT  teachers  had  access  to  “normal  instructional  materials”  and 
technological  resources  provided  by  the  districts.  Each  AZCOTT  teacher  received  the  materials  developed  for 
Educational  Media  and  Computers  598  taught  at  the  partner  university,  a copy  of  Teaching  with  Technology: 
Creating  Student-Centered  Classrooms  and  National  Educational  Technology  Standards  for  Students: 
Connecting  Curriculum  and  Technology,  and  an  increase  in  access  to  computers  for  their  students  through  one 
of  the  three  strategies  described  above. 

Procedures 

The  AZCOTT  program  operated  without  modification  during  the  2003-2004  school  year.  Each 
AZCOTT  teacher  participated  in  the  60  hours  of  professional  development  training  and  completed  the 
assignments.  Both  AZCOTT  and  Non-AZCOTT  teachers  participated  in  other  professional  development 
opportunities  provided  by  the  district  or  other  entities. 

Evaluation  Measures 

The  primary  evaluation  measures  used  in  this  study  were:  l)Teacher  Technology  Use  Questionnaire, 
2)Student  Technology  Skills  Performance  Assessment,  3)Student  Technology  Skills  Questionnaire,  4)  Student 
Technology  Activities  Survey,  5)Teacher  Program  Evaluation  Survey,  and  6)  Teacher  Interviews. 

The  Teacher  Technology  Use  Questionnaire  was  administered  to  the  AZCOTT  teachers  and  Non- 
AZCOTT  teachers  in  May  2004.  The  questionnaire  addressed  the  amount  of  time  a teacher  used  a computer  for 
planning  and  instruction  during  the  school  year,  the  types  of  professional  tasks  performed  on  the  computer,  the 
amount  of  time  their  students  used  computers,  and  the  types  of  technology  activities  participated  in  by  their 
students. 

The  Student  Technology  Skills  Performance  Assessment  was  administered  in  May  2004.  Ten  students 
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were  randomly  selected  from  each  participating  teacher’s  classroom  to  complete  the  assessment.  The 
assessment  consisted  of  asking  students  to  perform  18  steps  required  to  produce  and  modify  a word  processing 
document.  The  skills  needed  for  this  performance  were  selected  from  the  Arizona  Technology  Education 
Standards  for  grades  3-8  (Arizona  Department  of  Education,  2000).  A pilot  of  the  assessment  was  conducted 
with  25  fourth  graders  and  some  wording  was  modified  to  increase  clarity.  The  assessment  was  completed  in  a 
computer  lab  and  printed  as  the  final  step.  No  explanations  of  the  steps  were  provided  to  the  students.  The 
assessments  were  scored  on  a 1-0  basis,  1 indicating  that  the  step  was  performed  correctly  on  the  student’s 
document  and  0 indicating  that  it  was  not.  All  performance  assessments  were  scored  by  the  evaluator  without 
knowledge  of  the  student’s  group. 

AZCOTT  and  Non-AZCOTT  students  completed  the  Student  Technology  Skills  Questionnaire  in  May 
2004.  The  questionnaire  addressed  how  well  they  could  perform  a total  of  21  different  tasks  on  the  computer. 

In  addition,  AZCOTT  and  Non-AZCOTT  students  completed  a Student  Technology  Activities  Survey 
in  May  2004.  The  survey  addressed  how  frequently  they  performed  15  different  tasks  on  the  computer  during 
the  past  school  year. 

All  17  AZCOTT  teachers  completed  a Teacher  Program  Evaluation  Survey  at  the  end  of  the  last 
professional  development  session  in  May  2004.  The  20  items  addressed  the  overall  effectiveness  of  the 
AZCOTT  program,  the  program’s  impact  on  teaching  and  learning,  the  support  provided  to  the  teachers, 
unexpected  outcomes,  and  the  program’s  most  and  least  effective  elements. 

Two  AZCOTT  teachers  from  each  of  the  five  participating  districts  were  randomly  selected  to  be 
interviewed.  The  interviews  occurred  either  in  person  or  by  telephone.  The  interview  protocol  consisted  of  five 
open-ended  items.  The  interviews  averaged  about  15  minutes  in  length. 

Results 

The  results  are  reported  in  this  section  for  AZCOTT  and  Non-AZCOTT  teacher  reports  of  their  use 
and  student  use  of  technology  and  related  activities,  the  performance  assessment  of  students  on  computer- 
related  skills,  student  self-reports  of  their  computer  skills,  and  student  self-reports  of  their  computer  activities  in 
the  classroom.  Student  selection  of  appropriate  software  tools  and  teacher  attitudes  towards  the  AZCOTT 
program  are  also  reported. 

Teacher  Reports  of  Teacher  and  Student  Technology  Use 

Data  from  the  Spring  2004  Teacher  Technology  Use  Questionnaire  are  reported  in  Table  1. 

The  table  reveals  that  nine  of  the  17  AZCOTT  teachers,  but  only  two  of  the  15  Non-AZCOTT  teachers, 
reported  using  computers  more  than  60  minutes  a week  to  deliver  instruction.  Similarly,  ten  AZCOTT  teachers, 
but  only  two  Non-AZCOTT  teachers  reported  that  their  students  use  computers  more  than  60  minutes  a week  in 
the  classroom. 

Results  were  quite  similar  for  the  AZCOTT  and  Non-AZCOTT  teachers  on  the  six  professional  tasks 
listed  in  Table  1,  with  two  exceptions.  Fifteen  of  the  17  AZCOTT  teachers,  but  only  eight  of  the  15  Non- 
AZCOTT  teachers,  reported  using  computers  to  make  student  handouts  and  13  AZCOTT  teachers,  compared  to 
eight  Non-AZCOTT  teachers,  reported  using  the  computer  at  least  once  a week  to  get  information  from  the 
Internet  for  lessons. 

AZCOTT  teachers  also  reported  their  students  used  computers  for  more  time  than  Non-AZCOTT 
teachers  for  each  of  the  seven  student  activities  in  Table  1.  The  largest  differences  in  frequencies  were  for 
“Searching  for  information’’  (13  of  the  17  AZCOTT  teachers  reported  once  a week  or  more,  but  only  one  of  15 
Non-AZCOTT  teachers),  “Producing  multimedia  presentations”  (7  AZCOTT  teachers  reported  once  a week  or 
more,  but  only  one  Non-AZCOTT  teacher). 
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Table  1 Spring  2004  AZCOTT  and  Non-AZCOTT  Teacher  Survey  Frequencies 
AZCOTT  N=17  Non-AZCOTT  N=  15 


AZCOTT 

Non-AZCOTT 

Item 

60+min/wk 

< 60  min/wk 

60+min/wk 

< 60  min/wk 

Minutes  per  week  teachers  use  computers  to 
deliver  instruction 

9 

8 

2 

13 

How  often  during  a week  do  you  use 
computers  for  these  tasks? 

about  once/wk 

< once/wk 

about  once/wk 

< once/wk 

Record  or  calculate  student  grades 

16 

1 

12 

3 

Make  handouts  for  students 

15 

2 

7 

8 

Write  lesson  plans 

17 

0 

15 

0 

Get  information  from  the  Internet  for  lessons 

13 

4 

8 

7 

Exchange  files  with  other  teachers 
electronically 

4 

13 

1 

14 

Participate  in  discussion  boards,  listservs,  etc 

16 

1 

13 

2 

Minutes  per  week  students  use  computers  in 

60+min/wk 

<60  min/wk 

60+min/wk 

<60  min/wk 

the  classroom 

10 

7 

2 

13 

How  many  minutes  per  week  do  your 
students  spend  using  computers  for  these 
types  of  activities? 

30+min/wk 

<30  min/wk 

30+min/wk 

<30  min/wk 

Composing  (no  paper  involved) 

7 

10 

4 

11 

Publishing  written  work  (drafted  on  paper) 

9 

8 

6 

9 

Communicating 

2 

15 

1 

14 

Searching  for  information 

13 

4 

1 

14 

Producing  multimedia  presentations 

7 

10 

1 

14 

Organizing  information/planning 

4 

13 

0 

15 

Practicing  computer  skills 

7 

10 

2 

13 
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Assessment  of  Student  Performance 

The  18  steps  of  the  Student  Technology  Skills  Performance  Assessment  were  ordered  based  on 
predicted  difficulty  using  the  Arizona  Technology  Education  Standards  as  the  guide.  The  skills  from  the  Student 
Technology  Skills  Performance  Assessment  are  shown  in  rank  order  by  the  performance  of  AZCOTT  students 
in  Table  2. 

The  table  reveals  that  the  total  mean  scores  across  the  18  skills  were  12.28  for  the  AZCOTT  students  and  10.29 
for  the  Non-AZCOTT  students.  A one-way  analysis  of  variance  (ANOVA)  conducted  to  test  the  overall  mean 
scores  of  the  AZCOTT  and  Non-AZCOTT  students  for  significance  revealed  that  the  mean  of  the  AZCOTT 
students  was  significantly  higher  than  that  of  their  Non-AZCOTT  counterparts,  F(2,382)  = 7.64,  pc. 001. 

Table  2 AZCOTT  and  Non-AZCOTT  Student  Performance  Assessment  Mean  Scores  By  Skill 
AZCOTT  N=195  Non-AZCOTT  N=  187 


1.0= 

highest  possible  mean  for  each  skill  and  0 lowest  possible  mean  for  each  skill 

Skill 

AZCOTT 

Non-AZCOTT 

Overall 

1 

Open  your  word  processing  program 

1.00 

.82 

.91 

2 

Type  the  title  of  your  story 

.98 

.98 

.98 

3 

Make  the  title  font  size  36 

.93 

.83 

.88 

4 

Type  three  sentences 

.87 

.82 

.84 

5 

Type  your  grade  level  under  the  title 

.79 

.81 

.80 

6 

Center  the  title 

.77 

.72 

.75 

7 

Underline  the  title 

.69 

.68 

.69 

8 

Add  a piece  of  clipart  to  your  document 

.69 

.51 

.60 

9 

Make  a copy  of  the  clipart  and  paste  it  in  your  document 

.55 

.42 

.49 

10 

Enter  two  blank  lines  between  the  title  and  three  sentences 

.55 

.26 

.41 

11 

Go  on  the  Internet 

.54 

.44 

.49 

12 

Go  to  the  web  site  www.zoo.org 

.54 

.44 

.49 

13 

Make  one  of  the  two  clipart  pictures  bigger 

.53 

.38 

.46 

14 

Copy  a picture  from  the  web  site  and  paste  it  into  your  story 

.52 

.42 

.47 

15 

Draw  a smiley  face 

.44 

.36 

.40 

16 

Copy  and  paste  web  address  into  the  document 

.41 

.28 

.35 

17 

Copy  and  paste  the  web  address  under  the  picture 

.33 

.24 

.28 

18 

Change  the  top  margin  to  2.0  inches 

.14 

.13 

.13 

Total  Score 

*12.28 

10.29 

11.31 

18.0  = maximum  score  (1.0  per  skill) 

* AZCOTT  students  scored  significantly  higher  (pc. 001)  than  Non-AZCOTT  students  on  the  total  score. 

Table  2 also  reveals  the  AZCOTT  students  scored  higher  on  16  of  the  18  individual  skills  in  the 
performance  with  one  additional  skill  being  a tie.  Non-AZCOTT  students  scored  higher  on  only  one  skill. 

The  individual  skills  in  Table  2 are  listed  in  rank-order  by  AZCOTT  performance  from  highest  to  lowest. 
Comparison  of  the  observed  difficulty  of  these  18  skills  with  the  order  of  difficulty  in  which  they  appear  in  the 
Arizona  Technology  Education  Standards  yielded  a significant  (pc. 001)  Spearman  rank-order  correlation  of  .92, 
indicating  a very  high  correlation  between  their  order  of  difficulty  in  the  state  standards  and  the  order  obtained 
by  this  study. 
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Student  Reports  of  Technology  Skills 

The  percentage  of  AZCOTT  and  Non-AZCOTT  students  choosing  each  response  on  the  Student 
Technology  Skills  Questionnaire  is  reported  in  Table  3.  The  table  reveals  that  the  mean  percentage  score  across 
the  21  skills  for  students  who  reported  that  they  can  do  the  skills  well  was  40%  for  the  AZCOTT  students  and 
28%  for  the  Non-AZCOTT  students.  In  contrast,  the  means  for  “Cannot  do  it”  were  32%  for  the  AZCOTT 
students  and  45%  for  the  Non-AZCOTT  students.  The  AZCOTT  students  had  a higher  percentage  score  than 
the  Non-AZCOTT  students  on  all  21  items.  They  also  scored  20%  or  more  higher  in  the  “Can  do  it  well” 
category  on  seven  items:  write  a story,  letter  or  report;  change  font  size  and  style;  save  a copy  of  a document 
with  a new  name;  add  clipart;  change  the  size  of  a picture;  copy  pictures  from  the  Internet  into  a document;  and 
make  a multimedia  presentation. 

Table  3 Spring  2004  AZCOTT  and  Non-AZCOTT  Student  Technology  Skills  Reports  in  Percentages 
AZCOTT  N=401  Non-AZCOTT  N=  240 

Student  Reports  of  Technology  Activities 


Item 

AZCOTT 

Can  „ 

, . Can 

doit  , • 

do  it 

well 

Can 
not  do 
it 

Non-AZCOTT 

Can  „ 

, . Can 

do  it  , 

do  it 

well 

Can 
not  do 
it 

1 

Write  a story,  letter  or  report 

61 

35 

4 

40 

51 

9 

2 

Copy  and  paste  text 

48 

32 

20 

31 

25 

44 

3 

Change  tabs  and  margins 

31 

33 

36 

27 

27 

46 

4 

Change  font  size  and  style 

75 

19 

6 

55 

23 

22 

5 

Save  a copy  of  a document  with  a new  name 

68 

24 

8 

48 

23 

29 

6 

Add  clipart 

59 

26 

15 

38 

20 

42 

7 

Add  a digital  picture 

25 

31 

44 

15 

33 

52 

8 

Change  the  size  of  a picture 

67 

23 

10 

44 

35 

21 

9 

Copy  pictures  from  the  Internet  into  a document 

58 

24 

18 

35 

25 

40 

10 

Make  a spreadsheet 

20 

29 

51 

14 

31 

55 

11 

Create  a database 

8 

23 

69 

7 

14 

79 

12 

Make  a multimedia  presentation 

40 

27 

33 

17 

24 

59 

13 

Make  a web  page 

18 

26 

56 

15 

31 

54 

14 

Scan  a picture  and  insert  it  into  a document 

23 

23 

54 

19 

21 

60 

15 

Download  video  onto  a computer  and  edit  it  to 
make  a movie 

17 

27 

56 

13 

21 

66 

16 

Create  graphics/pictures 

32 

38 

30 

27 

35 

38 

17 

Send  email 

53 

22 

25 

45 

25 

30 

18 

Add  attachments 

30 

20 

42 

25 

23 

52 

19 

Write  a citation  for  an  electronic  source 

26 

30 

44 

7 

15 

78 

20 

Use  a digital  camera 

53 

29 

18 

41 

32 

27 

21 

Use  a projector  to  present  work 

35 

27 

38 

18 

30 

52 

Mean  Percentage  Score 

40 

27 

32 

28 

27 

45 
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Data  from  the  Student  Technology  Activities  Survey  reveal  that  of  the  15  activities  on  the  survey,  an 
average  of  39%  of  the  AZCOTT  students  and  29%  of  the  Non-AZCOTT  students  said  they  had  done  each 
activity  often  during  the  last  school  year.  The  five  activities  most  commonly  reported  as  being  done  “Often” 
versus  “Sometimes”  or  “Never”  by  AZCOTT  students  are:  “Researched  a topic  on  the  Internet”  (62%), 
“Learned  math”  (52%),  “Worked  with  other  students  to  create  projects”  (51%),  “Typed  or  published  a paper” 
(48%),  and  “Learned  language  arts”  (48%).  The  five  activities  most  commonly  reported  as  being  done  “Often” 
versus  “Sometimes”  or  “Never”  by  Non-AZCOTT  students  are:  “Took  a test”  (73%),  “Learned  math”  (48%), 
“Worked  with  other  students  to  create  projects”  (48%),  “Practiced  math  skills”  (40%),  and  “Researched  a topic 
on  the  Internet”  (30%). 


Student  Selection  of  Appropriate  Software  Tools 

The  last  section  of  the  Student  Technology  Activities  Survey  required  students  to  select  an  appropriate 
software  tool  to  use  for  a given  task.  For  six  of  the  seven  tasks  the  percentage  of  AZCOTT  students  that 
selected  the  “Correct”  tool  versus  the  “Incorrect”  or  responded  “I  don’t  know”  was  greater  than  for  the  Non- 
AZCOTT  students.  The  one  exception  was  “Draw  a picture”  with  AZCOTT  scoring  70%  correct  and  Non- 
AZCOTT  scoring  77%  correct.  The  AZCOTT  students  scored  20%  or  higher  more  than  the  Non-AZCOTT 
students  for  four  of  the  seven  tasks.  The  tasks  were:  make  an  idea  or  concept  map,  write  a story,  create  a 
presentation,  and  locate  information. 


Teacher  Attitudes 

Each  of  the  AZCOTT  teachers  completed  a 20-item  Program  Evaluation  Survey  consisting  of  17 
Likert  scale  items  and  three  open  ended  items  to  assess  teacher  attitudes  towards  the  program.  The  overall  mean 
was  3.1  for  the  17  Likert  items  on  a scale  from  4 (Strongly  Agree)  to  0 (Strongly  Disagree).  The  most  positive 
responses  were  for  “My  students  react  positively  to  technology -rich  classroom  activities.”  (M  = 3.9),  “I  use 
technology  more  with  my  students  this  year.”  (M  = 3.6),  “It  was  helpful  to  use  TaskStream®  to  create  my 
instructional  plans.”  (M  = 3.6),  and  “AZCOTT  was  a beneficial  professional  development  experience.”  (M  = 
3.5).  The  least  positive  responses  were  for  “Past  AZCOTT  teachers  provided  helpful  feedback.”  (M  = 1.8)  and 
“AZCOTT  was  responsible  for  me  taking  on  a leadership  role  this  year”  (M  = 2.0). 

Responses  to  the  open-ended  items  were  collected  and  categorized  by  common  themes.  The  most 
common  response  to  “What  were  the  most  effective  elements  of  the  AZCOTT  program?”  was  an  increase  in 
the  use  of  technology  due  to  of  access  to  additional  equipment,  which  was  mentioned  by  16  of  the  17  AZCOTT 
teachers.  The  most  frequent  response  (N=5)  to  “What  were  the  least  effective  elements  of  the  AZCOTT 
program”  was  “Creating  two  units  of  instruction,”  which  was  a requirement  of  the  AZCOTT  program. 

The  teacher  attitude  data  were  supplemented  by  information  obtained  in  individual  interviews  with  ten 
AZCOTT  teachers.  In  response  to  the  question,  “How  did  AZCOTT  impact  your  teaching?”  the  teachers 
reported  a variety  of  ways  the  program  influenced  their  teaching.  These  included  an  increase  in  their  efforts  to 
integrate  the  technology  into  their  teaching  (N=8)  and  making  their  instruction  more  learner  centered  (N=5). 
The  responses  to  “What  do  you  wish  you  had  learned?”  revealed  three  themes:  teachers  wanted  more  technical 
training  on  hardware  and  software,  more  ideas  for  managing  technology  use,  and  more  technology  integration 
strategies. 


Discussion 

This  evaluation  was  conducted  to  investigate  the  effect  of  the  Arizona  Classrooms  of  Tomorrow  Today 
program  on:  1)  the  types  and  frequency  of  digital  technology  activities  that  teachers  use  with  their  students  and 
that  students  perform  in  the  classroom,  2)  student  performance  on  a technology  skills  assessment  measure,  3) 
student  perceptions  of  their  technology  skills,  and  4)  student  reports  of  their  use  of  technology  for  classroom 
activities.  Other  factors  examined  included  AZCOTT  and  Non-AZCOTT  students’  ability  to  select  appropriate 
software  for  given  tasks  and  teacher  attitudes  toward  the  AZCOTT  program. 

The  AZCOTT  teachers  reported  using  computers  for  more  time  per  week  to  deliver  instruction  than 
Non-AZCOTT  teachers.  They  also  reported  using  computers  to  perform  professional  tasks  more  frequently  than 
the  Non-AZCOTT  teachers.  The  teacher  reports  also  indicated  that  AZCOTT  students  spent  considerably  more 
time  using  computers  in  the  classroom  than  Non-AZCOTT  students  and  that  the  time  was  distributed  across  a 
greater  variety  of  activities.  Thus,  based  on  teacher  reports  the  long-term  professional  development  training  and 
increase  in  technology  access  under  the  AZCOTT  program  had  the  desired  effect  of  increasing  both  teacher  use 


116 


of  technology  for  instructional  purposes  and  student  participation  in  technology-based  instructional  activities. 
Searching  for  information,  producing  multimedia  presentations  and  practicing  computer  skills  were  the  student 
activities  for  which  the  greatest  differences  were  reported  between  AZCOTT  and  Non-AZCOTT  students. 

The  reported  greater  overall  involvement  of  AZCOTT  students  with  technology  appears  to  be  the  most 
likely  reason  for  their  significantly  better  performance  than  Non-AZCOTT  students  on  the  performance 
measure.  The  overall  mean  of  the  AZCOTT  students  on  this  measure  was  approximately  two  points  (12.28  to 
10.29)  higher  than  that  of  the  Non-AZCOTT  students,  not  a huge  absolute  difference  but  one  that  was  highly 
significant  statistically  (pc. 001).  Furthermore,  the  fact  that  the  AZCOTT  students  scored  higher  on  16  of  the  18 
skills  comprising  the  performance  measure  indicates  that  their  participation  in  the  program  produced  quite  a 
consistent  effect  across  the  individual  skills. 

The  self-report  by  AZCOTT  and  Non-AZCOTT  students  on  their  technology  skills  is  consistent  with 
the  findings  from  both  the  teacher  questionnaire  and  the  performance  assessment.  AZCOTT  students  reported  a 
higher  ability  to  perform  all  21  activities  presented  in  Student  Technology  Skills  Questionnaire  than  did  the 
Non-AZCOTT  students.  Interestingly,  two  of  the  greatest  differences  favoring  AZCOTT  students  were  on  “big 
picture”  types  of  tasks:  write  a story,  letter  or  report  and  make  a multimedia  presentation.  The  greater  self- 
efficacy  for  computer  use  reflected  in  the  AZCOTT  students’  reports  may  indicate  they  are  more  likely  to 
choose  to  use  computers  at  school  or  home.  In  turn,  their  greater  current  skills  and  self-efficacy  could  contribute 
to  an  increase  in  continuing  motivation,  defined  as  one  freely  returning  to  a task  (Maehr,  1976),  to  use 
technology  and  an  increase  in  performance  in  the  future. 

Of  course,  higher  ability  to  perform  technology  skills  well  as  indicated  by  AZCOTT  students  on  both 
the  performance  measure  and  self-the  report  of  student  technology  skills,  should  be  associated  with  greater 
participation  in  technology  activities.  That  was  indeed  the  case  in  the  present  study.  An  average  of  39%  of 
AZCOTT  students,  but  only  29%  of  Non-AZCOTT  students  reported  that  they  had  participated  often  in  the  15 
computer-related  activities  in  the  survey.  Further,  the  activity  cited  “often”  most  frequently  by  AZCOTT 
students  was  “researched  a topic  on  the  Internet”  (62%  to  30%  for  Non-AZCOTT  students)  while  the  activity 
cited  as  “Often”  most  frequently  by  Non-AZOTT  students  was  “took  a test”  (73%  to  45%  for  AZCOTT 
students).  The  AZCOTT  professional  development  program  stresses  student-centered  uses  of  technology,  which 
may  have  been  be  a contributing  factor  to  this  difference  in  the  most  common  type  of  activity  between 
AZCOTT  and  Non-AZCOTT  students. 

The  data  on  student  selection  of  appropriate  software  tools  were  also  supportive  of  the  AZCOTT 
program.  The  fact  that  AZCOTT  students  scored  20%  or  higher  more  than  Non-AZCOTT  students  on  selecting 
the  appropriate  tools  for  making  an  idea  or  concept  map,  writing  a story,  creating  a presentation,  and  locating 
information,  was  particularly  impressive.  The  writing  a story,  creating  a presentation,  and  locating  information 
items  are  consistent  with  student  self-reports  showing  20%  or  greater  differences  between  AZCOTT  and  Non- 
AZCOTT  students  in  their  self-reported  technology  skills  for  the  first  two  items  and  in  their  frequent  classroom 
activities  for  the  “locating  information”  item. 

The  AZCOTT  program  was  a year-long  experience  that  involved  professional  development  training  in 
the  use  of  technology  for  instructional  purposes  as  well  as  increased  access  to  computers  in  the  classroom.  This 
evaluation  yielded  clear  evidence  that  the  teacher  and  student  use  of  computers  for  instructional  purposes, 
students’  computer  skills,  and  students’  perceptions  of  their  own  self-efficacy  with  regard  to  computer  use  were 
higher  for  those  that  participated  in  the  program  versus  the  comparison  group.  These  results,  while  not 
overpowering  in  their  effect,  were  consistently  positive  across  both  teacher  and  student  measures.  The  study 
indicates  that  a focused  longer-term  effort  to  increase  the  technology  training  of  teachers  and  access  to 
computers  for  their  students  can  lead  to  improvements  in  their  students’  computer  skills  and  in  their  use  of 
computers  to  enhance  their  learning  in  the  classroom. 
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Investigating  Student  Learning  in  a Constructivist  Multimedia -Rich 

Learning  Environment 

Doris  U.  Bolliger 
St.  Cloud  State  University 

Abstract 

It  has  been  suggested  in  the  existing  literature  that  the  use  of  constructivist  approaches  in  the 
educational  setting  contributes  to  active  learning  and  knowledge  transfer  for  students.  This  paper  provides  an 
overview  of  constructivist  approaches  used  in  a graduate-level  instructional  media  production  course  at  a mid- 
western  comprehensive  university.  Qualitative  data  collection  techniques  were  used  to  investigate  the 
perceptions  and  learning  of  students  in  an  environment  in  which  both  students  and  content  were  the  center  of 
the  learning  experience.  The  findings  suggest  the  use  of  active  learning  approaches,  in  which  students  have  the 
opportunity  to  interact  with  peers  and  the  instructor,  discussion  and  reflection  on  learning  experiences,  and 
encouragement  of  knowledge  sharing,  contribute  to  student  learning. 

Introduction 

Constructivism  has  gained  acceptance  and,  today,  is  highly  valued  by  many  educators.  Constructivism 
is  an  educational  theory  about  how  knowledge  is  acquired  and  how  individuals  learn  (Brooks  & Brooks,  1993). 
Constructivism  is  about  thinking  and  understanding.  Grounded  in  this  theory  is  that  individuals  obtain 
knowledge  by  creating  constructs  and  by  interpreting  and  reflecting  on  their  exp  eriences  (Jonassen,  Peck,  & 
Wilson,  1998).  Only  constructs,  which  can  be  a schema  or  a concept,  can  be  stored  when  we  process 
information. 

Central  to  this  idea  are  self -regulation,  active  learning,  individual  differences,  social  learning,  and 
reflection  (Gagnon  & Collay,  2001;  Jonassen,  Peck,  & Wilson,  1998).  Individuals  learn  by  investigating, 
discovering,  and  creating  structures;  actively  attaching  meaning  to  a concept;  and  integrating  new  or  modified 
constructs  into  existing  knowledge.  According  to  Novak  (1998),  meaningful  learning  involves  “thinking, 
feeling,  and  acting”  (p.  9).  Because  knowledge  construction  is  different  for  individuals,  learning  has  many 
varieties.  For  a constructivist  the  absolute  truth  cannot  be  verified. 

Constructivists  do  not  argue  that  constructivist  approaches  lead  to  learning  and  others  do  not. 
However,  Stahl  (2003)  points  out  that  everyone  is  a constructivist  because  we  are  constructing  knowldege  at 
every  conscious  moment.  Constructivists  share  the  opinion  that  students  have  been  passive  receivers  of 
knowledge  for  too  long.  Teachers  have  given  too  many  lectures  and  transmitted  their  knowledge  to  students  by 
acting  as  Sage  on  the  Stage.  Teachers  have  traditionally  been  viewed  as  experts,  God-like  individuals  who  are 
not  to  be  questioned.  Because  of  their  status,  they  are  located  on  the  top  of  a power  structure  which  places 
students  on  the  bottom  level.  Students  are  thought  of  as  empty  vessels  not  capable  of  thinking  for  themselves. 
They  study  instructional  material  by  memorizing  facts  but,  shortly  thereafter,  forget  what  they  have  learned. 
The  transfer  that  should  take  place  from  theory  to  practice  does  not  occur.  Instead  of  using  the  transmission 
approach  (Berge,  2001),  in  which  students  receive  knowledge  in  a top-down  delivery  system  from  teachers, 
constructivists  argue  students  should  be  actively  involved  in  their  learning  so  that  they  are  able  to  apply  what 
they  have  leaned. 

Learning  should  be  fun  and  not  dreaded  as  mindless  activity  where  students  end  up  reciting  facts  they 
stored  in  short -tem  memory.  Teachers  should  ask  how  they  can  provide  students  with  environments  in  which 
teachers  can  facilitate  student  learning  so  that  students  (a)  can  “discover,  create,  and  apply  knowledge  for 
themselves”,  (b)  “push  themselves",  and  (c)  “truly  understand  what  they  learn”  (Marlowe  & Page,  1998,  p.  5). 
According  to  Marlowe  and  Page,  components  that  need  revision  include  terminology,  communication  between 
involved  parties,  learning  activities,  learning  environments,  student  motivation,  and  student  assessment. 

Perceived  Barriers 

Instructors  who  desire  to  use  constructivist  approaches  or  have  used  them  in  their  classrooms  face 
barriers  from  administration,  peers,  and  students.  Administrators  are  concerned  that  the  integration  of 
constructivist  approaches  takes  too  much  time.  Learner-centered  activities  such  as  discussions  or  group  work 
take  more  time  than  lectures.  Because  of  accreditation  requirements  instructors  must  cover  content  instead  of 
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using  up  time  with  student-centered  approaches. 

The  roles  of  instructors  and  learners  shift  in  the  constructivist  environment.  Instructors  become 
mentors,  coaches,  and  facilitators.  It  requires  teachers  to  modify  existing  materials  and  activities.  Some  may 
oppose  constructivist  approaches  because  they  feel  this  process  is  too  difficult  and  time  consuming.  In  addition, 
they  might  not  be  thrilled  to  share  control  because  constructivists  typically  surrender  some  control  to  their 
students.  Instructors  who  see  themselves  as  the  experts  in  the  field  may  feel  threatened  if  students  start 
questioning  them  about  content  or  facts. 

Student  roles  also  change.  The  learner  now  becomes  responsible  for  his  or  her  own  learning.  Many 
students  have  been  trained  to  comply  with  course  requirements  and  have  not  been  taught  to  think  for 
themselves.  Teachers  who  would  like  to  implement  constructivist  approaches  may  find  that  students  resist  this 
change. 


The  Study 

The  purpose  of  this  study  was  to  determine  how  students  would  perceive  constructivist  approaches  in 
the  classroom  and  their  own  learning.  The  researcher  was  particularly  interested  in  (a)  how  easily  students 
would  adapt  to  the  approaches,  (b)  approaches  perceived  as  useful  by  students,  and  (c)  approaches  that  were  not 
effective. 


Methodology 

Setting 

The  study  was  undertaken  at  a public  comprehensive  university  with  approximately  16,000  students  in 
the  mid-west.  The  course  offered  during  spring  2003  was  a graduate  level,  computer-based  authoring  course  for 
individuals  who  majored  in  information  media.  Students  in  the  human  resources  development  and  training  track 
were  required  to  complete  the  course  successfully;  others  could  choose  the  course  as  an  elective.  The  instruction 
took  place  in  a classroom-based  environment,  but  was  enhanced  with  a Web-based  course  management  system 
and  other  Web  tools  such  as  html  files,  e-mail,  and  so  forth. 

The  classroom  in  which  the  course  was  held  was  located  in  a state -of-the  art  facility  and  was  equipped 
with  31  personal  computers,  projector,  and  a VCR.  Instructors  and  students  in  this  classroom  had  a variety  of 
software  programs  available  to  them  including,  but  not  limited  to,  Adobe  Photoshop;  Inspiration;  Macromedia 
Dreamweaver,  Fireworks,  and  Flash  MX;  Microsoft  Office,  Visio,  and  FrontPage;  and  Click2Learn  ToolBook 
Instructor.  The  facility  also  housed  several  open  computer  laboratories  allowing  students  access  to  printers, 
scanners,  digital  cameras,  and  laptop  computers. 

Participants 

Nine  students  were  enrolled  in  the  instructional  media  production  course.  Some  of  the  students 
attended  the  university  on  a full-time  basis;  others  were  part-time  students.  Fifty  percent  of  students  were 
employed  full-time  in  the  education  or  training  industry,  others  were  employed  on  a part-time  basis.  Students  in 
this  group  varied  greatly  on  distribution  of  age,  progress  made  in  their  program  and,  subsequently,  varied 
greatly  on  existing  computer  and  authoring  skills.  However,  all  students  in  the  course  had  successfully 
completed  an  instructional  design  course,  a prerequisite  for  the  course. 

Course  Introduction 

At  the  beginning  of  the  course,  learners  were  introduced  to  course  materials  and  course  requirements. 
The  instructor  made  it  clear  that  the  course  was  not  simply  owned  by  her.  Students  were  asked  to  consider  the 
course  to  be  “their”  course  because  they  had  a vested  interest  and  were  given  voting  rights.  For  example,  the 
course  syllabus  included  a sentence  informing  participants  that  late  assignments  would  not  be  accepted.  The 
instructor  explained  why  this  stipulation  was  in  the  syllabus  but  gave  students  an  option  to  vote  on  the  issue, 
and  students  voted  for  flexible  due  dates.  During  the  first  class  session,  students  introduced  themselves  and  took 
pictures  of  one  another  with  a digital  camera.  These  pictures  were  later  placed  on  the  course  Web  site. 

Course  Material  Description 

Course  information.  The  instructor  designed  a course  Web  site  that  included  a syllabus,  schedule, 
assignments,  and  resources.  Contact  information  for  all  participants  was  also  listed  on  the  site.  The  instructor 
supplemented  the  classroom-based  course  with  WebCT,  a Web-based  course  management  system.  A link  to  the 
course  Web  site  was  provided  there,  as  well  as  some  course  content  materials  such  as  PowerPoint  presentations 
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for  the  first  few  chapters  discussed  in  class,  help  notes  for  special  topics,  and  a syllabus.  Other  WebCT  tools 
utilized  were  communication  and  evaluation  tools.  The  instructor  composed  six  threaded  discussion  messages  to 
which  students  replied.  Everyone  in  the  course  utilized  the  e-mail  function.  Students  were  also  able  to  submit 
their  assignments  through  WebCT  and  view  their  grades  and  feedback  provided  by  the  instructor. 

Materials  and  tools.  Students  used  a variety  of  software  programs  which  included  Macromedia 
Dreamweaver,  Fireworks,  and  Flash;  the  CourseBuilder  extension  for  Dreamweaver;  Microsoft  PowerPoint, 
Word,  and  Visio.  The  two  textbooks  required  for  the  course  were  a multimedia  development  text  and  a 
Dreamweaver  MX  self-study  text.  In  addition,  the  instructor  provided  several  software -based  books  in  class. 
Other  chapters  from  instructional  design,  instructional  technology,  and  test  theory  were  assigned,  as  well  as 
several  current  articles  pertaining  to  Web-based  and  computer-based  education  and  training. 

Assignments  and  requirements.  Students  completed  two  types  of  assignments:  mandatory  or  optional 
assignments.  Mandatory  assignments  included  (a)  a Web  site  on  which  other  assignments  were  posted,  (b)  a 
computer-authored  instructional  product  for  a client,  (c)  a flowchart  and  storyboards,  (d)  a formative  peer 
evaluation,  (e)  a group  presentation,  (f)  and  class  participation.  Students  were  also  able  to  use  some  of  the  class 
time  for  working  on  their  projects. 

The  completion  of  mandatory  assignments  made  up  80%  of  the  student’s  grade.  Students  were  able  to 
choose  from  the  following  optional  assignments:  (a)  two  annotated  bibliographies,  (b)  an  image  editing  project, 
(c)  a Flash  MX  project,  (d)  a research  paper,  and  (e)  a final  examination. 

Jonassen  (2000)  emphasizes  the  importance  of  “grounded  educational  practices”  in  the  learner- 
centered  environment  (p.  11).  Instructions  for  assignments,  however,  were  kept  to  a minimum.  For  writing 
assignments,  the  instructor  purposely  did  not  include  any  parameters  such  as  paper  length,  research  topics,  and 
so  forth.  The  instructor  requested  students  to  submit  a proposal  outlining  the  purpose,  questions  students  sought 
to  explore,  and  a table  of  contents  in  order  to  provide  guidance  and  feedback  to  the  students  who  selected  these 
assignments.  Specifications  for  the  production  of  authoring  projects  were  also  limited  because  the  instructor  did 
not  want  to  limit  the  creativity  of  students.  During  the  course  of  the  semester,  group  members  were  required  to 
share  their  work  in  progress  with  the  class.  The  instructor  provided  feedback  to  the  groups  at  various  stages, 
particularly  once  they  completed  the  flowchart  and  storyboards.  In  order  to  provide  students  with  additional 
guidance  grading  rubrics  were  posted  on  the  Web  for  all  assignments. 

Activities.  Out-of-class  activities  included  generation  of  final  exam  questions,  posting  to  threaded 
discussions  in  WebCT,  reading  assigned  chapters  and  articles,  locating  resources  for  writing  assignments,  group 
work  related  to  the  client  project,  and  taking  the  final  examination.  In-class  activities  included  small  group  and 
whole  class  discussions  pertaining  to  assigned  readings,  threaded  discussions,  and  writing  assignments. 

Students  discussed  work  in  progress  and  shared  experiences.  They  presented  completed  assignments  to  the 
entire  class,  asked  questions  of  one  another,  and  provided  viewpoints  and  feedback  to  their  peers.  Some  class 
time  was  set  aside  to  work  on  all  elements  of  the  client  project. 

Instruction.  Lectures  were  kept  at  a minimum.  During  the  first  four  weeks,  the  Dreamweaver  sessions 
were  structured  like  hands-on  training  session.  Students  worked  through  chapters  covering  basic  skills  with  the 
instructor.  The  instructor  demonstrated  tasks  while  learners  repeated  the  exercises  on  their  computers.  One-on- 
one  assistance  was  available  from  the  instructor  and  a graduate  assistant.  Fireworks  and  Flash  sessions  were  less 
structured  demonstrations  because  these  assignments  were  optional.  Assigned  readings  were  discussed  in  either 
two  groups  or  with  the  entire  class.  Students  were  encouraged  to  ask  questions,  share  their  experiences,  express 
their  viewpoints,  differentiate  concepts,  and  critique  any  writings.  Only  when  students  could  not  answer 
questions  raised  during  the  discussions  did  the  instructor  provide  guidance  by  clarifying  points  and  concepts. 

Method 

The  instructor  and  a graduate  assistant  observed  students  during  the  class  sessions.  The  instructor 
initiated  discussions  regarding  the  assignments  and  tools  used.  The  students  were  asked  to  complete  a 3-minute 
evaluation  form  after  each  class  session  to  provide  feedback  to  the  instructor.  The  instructor  encouraged 
students  to  contact  her  with  any  questions  relating  to  the  course  and  provided  professional  and  personal  contact 
information  on  the  syllabus.  In  addition,  students  had  the  opportunity  to  contact  a graduate  assistant  who  was 
available  during  class  and  by  appointment.  The  graduate  assistant  kept  the  instructor  abreast  of  students  who 
sought  his  assistance. 

In  addition,  students  were  asked  to  provide  feedback  about  the  course  during  a short  interview  session. 
Participants  were  informed  that  the  short  session  was  not  a course  or  instructor  evaluation  and  that  the  purpose 
of  the  interview  was  not  to  gather  positive  feedback.  Rather,  the  interviewer  was  interested  in  ascertaining 
strategies  and  activities  that  helped  the  student  learn.  The  question  was:  What  activities  have  helped  you  learn 
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the  materials  in  this  course?  After  students  responded  to  this  question,  they  were  asked  to  complete  a 
questionnaire  with  a listing  of  specific  course  elements  and  strategies.  Individuals  indicated  which  elements 
were  or  were  not  helpful  and  identified  the  five  most  helpful  activities. 

Results 

In-Class  Observation 

Students  voiced  confusion  during  several  class  sessions.  They  were  not  accustomed  to  having  decision- 
making abilities  pertaining  to  course  structure  such  as  deadlines,  time  management,  and  assignment  parameters. 
The  limited  information  about  assignments  particularly  confused  students  even  though  online  grading  rubrics 
were  provided  on  the  course  Web  site.  The  graduate  assistant  who  observed  the  same  behavior  on  several 
occasions  confirmed  this  perception. 

Students  at  first  did  not  actively  take  responsibility  or  ownership  for  their  work.  For  example,  they  did 
not  ask  questions  about  the  first  annotation  and  did  not  follow  instructions  on  the  Web  site.  When  feedback 
about  this  optional  assignment  was  provided  by  the  instructor,  several  students  were  surprised  by  the 
instructor’s  expectations.  Before  that  class  session  began,  the  instructor  posted  an  example  on  the  Web  to 
provide  additional  guidance. 

In  one  instance,  students  inquired  about  the  required  length  of  the  research  paper.  The  instructor  in  turn 
asked  them  how  long  they  thought  the  paper  should  be.  Students  turned  to  each  other  in  disbelief.  It  appeared 
they  were  out  of  their  comfort  zone  regarding  this  experience.  A discussion  followed,  and  participants  decided 
approximately  ten  pages  were  appropriate.  The  instructor  indicated  any  length  was  acceptable  for  as  long  as 
they  met  all  requirements  listed  in  the  grading  rubric. 

Another  element  students  were  not  used  to  was  that  they  could  select  some  optional  assignments  and 
decide  which  topics  they  would  like  to  explore.  Several  times  during  class  sessions  students  said  that  they  were 
used  to  being  told  what  to  do.  As  the  semester  progressed  students  became  less  confused  and  took  responsibility 
for  their  own  learning;  it  appeared  that  they  enjoyed  working  on  their  assignments.  Not  surprisingly,  students 
selected  different  combinations  of  assignments,  chose  a wide  variety  of  topics,  and  used  several  different  tools. 
For  example,  tools  used  by  students  in  designing  their  Flowcharts  were  Word,  Visio,  or  PowerPoint;  each  group 
used  a different  tool. 

The  instructor  was  perplexed  that  students  were  confused.  She  expected  students  would  access  the 
assignment  information  on  the  Web.  Surprisingly,  not  all  students  accessed  this  information.  The  instructor 
needed  to  refer  them  to  the  Web  site  on  several  occasions  when  questions  pertaining  to  assignments,  grading, 
and  scheduling  were  raised. 

Another  concern  was  if  students  were  actually  learning  or  not.  Without  the  use  of  quizzes  and  tests,  the 
instructor  was  not  certain  if  students  were  learning  in  the  beginning  of  the  course.  However,  a few  weeks  into 
the  semester  students  conveyed  content  knowledge  and  understanding  during  discussions.  They  demonstrated 
that  they  mastered  new  skills  when  they  submitted  completed  assignments.  Students  actively  participated  in 
class.  They  were  asking  many  questions,  shared  their  experiences  and  viewpoints,  and  assisted  one  another 
during  class  sessions. 

Occasionally,  flexible  deadlines  caused  scheduling  problems.  Discussions  about  threaded  discussions 
and  show  & tells  needed  to  be  postponed  several  times  due  to  not  everyone  having  completed  the  assignment  on 
the  proposed  deadline.  The  instructor  needed  to  be  flexible  and  adjusted  the  schedule  accordingly.  Flexibility 
was  also  required  in  regard  to  the  use  of  class  time.  At  times,  students  were  so  engaged  in  discussions  that  other 
activities  needed  to  be  either  eliminated  or  rescheduled.  This  structured  chaos  in  the  classroom  was  responsible 
for  some  excellent  sessions  in  which  information  was  truly  shared  and  knowledge  individually  constructed. 

One  other  concern  was  the  rating  of  course  evaluations.  The  instructor  was  untenured  and  held  a 
probationary  position.  Because  the  introduction  of  change  can  produce  a level  of  dissatisfaction,  the  instructor 
was  concerned  about  the  possibility  of  receiving  unfavorable  ratings.  Administration  takes  course  evaluations 
seriously  at  this  university  and  results  are  used  in  the  renewal,  promotion,  and  tenure  decisions.  This  concern 
was  one  of  the  reasons  why  students  were  asked  to  complete  a 3 -minute  evaluation  at  the  end  of  each  session. 
When  students  expressed  concern  about  a particular  class  session,  the  instructor  was  able  to  address  the  issue  in 
the  beginning  of  the  next  class  session.  In  addition,  the  short  evaluation  form  provided  the  instructor  with 
valuable  feedback,  which  was  used  to  make  changes  throughout  the  semester. 

Interview  Responses 

When  asked  what  helped  students  learn,  seven  of  them  reported  the  hands-on  activities  were  helpful  to 
them  because  they  “learned  by  doing.”  The  same  number  of  students  pointed  out  the  in-class  discussions  helped 
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them  learn.  Interview  participants  clarified  the  interaction  was  good  and  they  appreciated  discussions  about 
chapters  in  the  Multimedia  textbooks.  One  student  indicated  the  class  discussions  “pulled  it  all  together”.  These 
students  also  enjoyed  listening  to  others’  viewpoints  and  found  it  helpful  to  hear  what  other  groups  working  on 
the  client  project  were  going  through. 

Six  participants  mentioned  that  the  group  work  on  the  project  helped  them  in  their  learning  process. 
Students  mentioned  the  perspectives  of  other  group  members  were  particularly  helpful,  and  they  were  able  to 
balance  the  workload  between  group  members.  Five  students  considered  the  workshop-style  Dreamweaver 
sessions  held  during  the  first  few  weeks  in  the  semester  helpful.  Two  students  each  mentioned  that  the 
following  elements  were  advantageous  to  them  learning  in  the  course:  (a)  the  Web  project,  (b)  Multimedia 
textbook,  (c)  assistance  of  the  graduate  assistant,  (d)  threaded  discussions,  (e)  annotations,  (f)  assigned  articles, 
(g)  class  schedule,  and  (h)  feedback.  Even  though  students  were  not  asked  which  elements  were  not  helpful  in 
their  learning  in  the  interview,  four  of  them  shared  some  of  these  elements  with  the  interviewer.  They  were  the 
course  management  system,  the  Flash  demonstration,  the  Dreamweaver  textbook,  and  threaded  discussions. 

Survey  Responses 

Activities  considered  helpful.  All  participants  indicated  the  following  course  activities  had  been  helpful 
in  their  learning:  (a)  in-class  discussions  in  small  groups  and  as  a whole,  (b)  showing  and  viewing  completed 
assignments,  (c)  completing  a research  paper  draft,  (d)  designing  a personal  Web  page,  and  (e)  working  on  all 
parts  of  the  client  project  (proposal,  outline,  flowchart,  storyboards,  and  the  product  itself),  (f)  providing  and 
receiving  feedback  during  a formative  evaluation,  and  (g)  presenting  the  final  group  project  to  the  class.  In 
addition,  all  students  agreed  (a)  flexible  due  dates,  (b)  online  grading  rubrics,  (c)  the  freedom  to  select  topics  for 
assignments,  (d)  resources  such  as  example  forms  posted  online,  and  (e)  instructor  feedback  helped  them  learn. 

Activities  with  the  highest  ratings.  Students  assigned  the  highest  ratings  to  the  following  course 
elements:  (1)  Web  project,  (2)  hands-on  activities,  (3)  group  work,  (4)  instructor  feedback,  (5)  group  discussion, 
(6)  multimedia  textbook,  (7)  selecting  assignments,  (8)  client  proposal,  and  (9)  client  project. 

Activities  not  considered  helpful.  One  activity  not  considered  helpful  by  the  majority  of  the  students 
(more  than  50%)  was  reading  assigned  chapters  in  the  Dreamweaver  textbook.  A large  percentage  of  students 
(44.4%)  did  not  consider  the  threaded  discussions  helpful,  and  33.3%  did  not  consider  the  image  manipulation 
project  with  Fireworks,  the  final  examination,  and  “our”  course  attitude  as  valuable  in  their  learning  process. 

Discussion  and  Implications 

The  introduction  of  constructivist  approaches  in  a classroom  with  learners  who  are  not  accustomed  to 
taking  responsibility  for  critical  thinking  and  learning  is  difficult.  The  instructor  must  truly  believe  in  this  theory 
in  order  to  continue  this  effort  because  of  the  barriers  encountered  by  various  constituencies.  Instructors  must  be 
flexible  to  accommodate  progress,  or  lack  thereof,  with  course  content  and  requirements. 

Many  students  have  been  taught  to  comply  with  what  their  instructors  tell  them  without  questioning 
the  experts.  Critical  thinking  and  reflection  can  be  learned,  however.  If  they  have  not  learned  these  skills  by  the 
time  they  arrive  in  our  classrooms,  we  should  strive  to  teach  them  these  skills,  because  they  will  need  them  once 
they  graduate  with  their  college  degrees. 

It  is  not  surprising  to  find  students  reported  hands-on  activities  and  discussions,  may  they  be  group 
discussions  or  exchanges  during  which  they  share  their  viewpoints,  were  helpful  in  their  learning.  Placing 
content  and  learners  in  the  center  of  the  learning  experience  by  engaging  students  in  the  learning  process,  giving 
them  the  opportunity  to  take  ownership  of  ideas  and  products,  and  providing  them  with  a learning  environment 
in  which  expression  and  reflection,  enables  them  to  form  constructs. 

Feedback  is  critical  in  student  learning.  Feedback  from  not  only  the  instructors  but  also  from  peers  is 
imperative  in  the  learning  process.  Learning  does  not  occur  in  a vacuum;  it  is  truly  a social  process.  Our 
students  are  not  empty  vessels  when  they  arrive  on  campus.  They  have  acquired  knowledge  elsewhere  and  had 
prior  life  and  professional  experiences  they  can  share  with  others.  Instructors  should  provide  students  with  the 
opportunity  to  revise  projects  and  learn  from  mistakes  in  order  to  facilitate  improvement.  Good  writers  do  not 
write  by  themselves  - professionals  use  a peer-review  process.  It  is  also  advantageous  for  students  to  build  good 
team  working  and  communication  skills.  Many  projects  in  the  business  industry  are  designed  and  produced  by  a 
team  of  individuals  utilizing  the  expertise  of  its  members. 

Conclusion 

The  researcher  hypothesized  some  of  the  course  activities  would  be  more  helpful  to  learners  than 
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others.  In  fact,  the  researcher  expected  that  certain  activities  would  be  clearly  identified  by  all  students.  Indeed, 
some  of  the  activities  that  were  expected  to  be  rated  highly  by  students  in  being  helpful  in  their  learning  process 
were  identified  as  such.  However,  there  were  a wide  variety  of  activities  that  received  high  ratings.  These 
results  indicate  instructors  should  design  a wide  variety  of  activities  and  assignments  in  order  to  support  student 
learning.  Because  not  all  students  learn  the  same  way,  we  need  to  take  individual  differences  and  learning  styles 
into  account.  This  approach,  however,  is  more  labor  intensive  for  the  instructor. 

Another  hypothesis  was  that  the  “our”  course  attitude  would  be  a successful  approach  in  the  course. 
The  instructor  expected  this  approach  would  set  the  stage  for  a relaxed  and  supportive  learning  environment. 
Students  did  not  report  that  this  approach  was  considered  helpful  in  their  learning.  One  student  wrote  on  the 
survey,  “Graduate  students  have  learned  to  do  what  they  are  told  to  do  so  this  part  is  difficult  to  get  used  to.” 
Perhaps  students  in  this  course  were  not  quite  prepared  to  encounter  this  type  of  learning  environment. 

Prepared  or  not,  we  should  provide  students  with  a safe,  supportive  environment  because  some  already 
experience  a high  level  of  stress  while  they  attend  universities.  Fear  of  failure  and  lack  of  control  and  power  is 
the  reality  of  many  students  in  higher  education  settings.  We  should  create  teachable  moments  by  creating 
supportive  environments  in  which  we  can  assist  learners  in  creating  constructs  and  internalizing  them  with  the 
goal  to  increase  retention  and  transferability  so  that  students  can  maximize  application. 

Readers  must  be  careful  in  generalizing  findings  in  this  study  to  other  populations.  The  study  involved 
a small  sample  of  graduate  students  at  one  comprehensive  university  in  the  mid  west.  There  is  a need  for 
replication  of  the  study  with  other  populations  and  a larger  sample  size. 
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Who’s  In  Charge? 

A System  of  Scaffolds  That  Encourages  Online 
Learners  to  Take  Control 

Naomi  R.  Boyer 
University  of  South  Florida 

Abstract 

The  teaching  and  learning  “dance”  is  one  that  traditionally  has  been  led  by  the  instructor  with  the 
student  following  passively.  Faculty  members  in  higher  education  are  entering  the  realm  of  online  learning , 
many  with  the  previous  experience  and  hopes  of  facilitating  student -centered,  active  learning  experiences. 
However,  due  to  factors  that  are  integral  to  this  environment,  many  are  returning  to  their  “comfort  zones”  by 
providing  greater  clarity  and  specificity,  stricter  accountability  measures,  and  less  student 
flexibility/personalization.  To  address  best  teaching  practices  in  adult  education  within  the  online 
environment,  a systems  model  of  social,  self-direction  is  presented  that  allows  the  student  to  “lead”  and  yet 
learn  how  to  facilitate  the  self-direction  process.  This  model  was  used  as  an  instructional  intervention  in  this 
study,  which  sought  to  answer  the  question:  “What  are  the  self-perceived  learning  gains  of  students  engaged  in 
a social,  self-directed  learning  experience?”  A self-rated  pre-test/post-test  design  was  utilized  with  the  8 course 
sections  and  112  subjects  that  received  this  instructional  intervention.  Other  data  sources  were  also  utilized  as 
triangulation  for  validating  the  self-reported  learning  gains  on  both  the  breadth  and  depth  of  course  material. 
The  model  was  found  to  facilitate  significant  learning  gains,  while  attending  to  university  guidelines  and  course 
requirements.  Further  implications  and  questions  that  are  resulting  from  this  research  are  also  explored. 

Introduction 

Online  learning  can  be  overwhelming  and  discombobulating  for  learners  due  to  uncertainty  and  lack  of 
clearly  expressed  expectation.  However,  in  response  to  student  requests  for  clarity  in  online  environments, 
structure  has  been  created  by  assuming  a more  rigid  classroom  approach  that  eliminates  many  of  the  benefits  of 
virtual  instruction.  Rather  than  assuming  new  and  innovative  ways  to  respond  to  demands  of  online  education, 
the  trend  is  to  assign  quotas,  dates,  and  accountability  measures  that  minimize  choice  and  encourage  students  to 
become  the  type  of  students  that  Ponticell  and  Zapeda  (2004)  term  “compliant  learners”.  This  was  substantiated 
in  many  presentations  at  a recent  national  leadership  conference  where  faculty  shared  innovations  in  program 
development  and  course  delivery  via  online  environments. 

Traditional  approaches  of  lecture,  readings,  and  testing  do  not  successfully  accommodate  the  best 
practices  of  higher  education/adult  education,  which  encourage  active,  engaged,  and  authentic  le  arning 
experiences.  Knowles’  work  provides  a definition  of  adragogy  and  self-direction  that  can  be  used  as  a 
theoretical  basis  for  incorporating  adult  learning  principles  into  higher  educational  teaching  practice  (Knowles, 
1975;  Knowles,  1986;  Knowles,  Holton,  & Swanson,  1998).  The  attributive,  representative,  and  situational 
theoretical  philosophies  have  been  posited  within  the  adult  education  field  as  different  yet  critical  teaching  and 
learning  perspectives  and  are  usually  explored  as  divergent  instructional  methods.  The  model  presented  in  this 
research  integrates  all  three  models  to  attend  to  learner  characteristics  (input  attributive  variables),  process  and 
meaning  construction  (process  representative  variables),  and  socially  contextual  interaction  (process  learning 
community,  self  and  group  metacognition,  and  outcome  environmental  variables)  (McGough,  2003).  While 
these  practices  are  important  regardless  of  educational  delivery,  the  advent  of  online  mediums  has  provided  a 
platform  for  the  exploration  of  innovative  teaching  models  and  an  adaptation  of  “instructor”  and/or  “student” 
roles  (Harvey,  2002;  Jonassen,  2002;  Moller,  2002). 

Aligning  instructional  approaches  so  that  online  experiences  provide  both  clearly  expressed  structure 
and  a means  for  personal  learning  that  incorporates  self-direction,  metacognition,  and  learning  communities  is 
not  an  easy  linear  task.  Instead,  learning  in  this  framework  must  be  viewed  as  a complex  system  where  students 
are  granted  responsibility  for  planning,  searching,  finding  and  producing  learning  objectives,  while  instructors 
provide  the  scaffolds,  resources,  feedback,  and  expertise  that  is  essential  to  connect  system  components.  Within 
this  framework,  learning  becomes  an  instructional  dance,  where  students  lead  movements,  direction,  and  pace 
while  instructors  follow  in  step,  provide  assistance,  and  enhance  the  experience.  So  the  question,  “Who’s  in 
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charge?”  becomes  difficult  to  discern  as  the  swirling  dance  of  learning  is  in  progress,  and  one  must  wonder  in 
this  design  whether  the  complexity  is  an  exercise  in  futility,  or  whether  breadth  and  depth  of  learning  does 
occur. 

Background 

A model  has  been  under  design  that  provides  both  instructional  technique  and  cognitive  theory  to 
accommodate  the  difficulties  of  structure  provision  and  personal/group  direction  of  learning.  (See  Figure  1). 

The  systems  model  of  social,  self-directed  learning  considers  input,  process,  and  output,  which  result  in 
outcome  variables  and  is  represented  by  the  formula  (I+P+0=Ou)  (Boyer,  2003).  In  this  model,  it  is 
hypothesized  that  a student  enters  a learning  situation  with  a given  set  of  input  variables  which  include  learning 
patterns,  previous  experience,  content  knowledge,  personal  interests,  and  a host  of  other  characteristics.  These 
input  variables  serve  to  mediate  the  instructional  process  that  is  designed  for  students  in  the  online  environment. 

The  process  portion  of  the  model  is  centered  on  individual  and  group  metacognition,  which  is 
surrounded  by  the  course  learning  communities.  The  instructional  design  and  components  then  enclose  the 
learning  communities.  These  instructional  design  components  include:  a self-directed  learning  framework  (acts 
as  a scaffold  for  students),  online  learning  tasks,  reflection,  interaction  and  engagement,  continuous  feedback, 
and  integrated  authentic  learning.  The  process  then  gives  way  to  the  output  portion  of  the  system,  which 
includes  in  this  case,  overall  learning  gain  (content  specific),  final  products,  course  evaluations,  group 
effectiveness,  couise  completion  rates,  and  real  world  authentication.  It  is  assumed  that  the  model  includes  a 
feedback  loop  that  continues  to  drive  the  overall  learning  system. 

The  model  of  social,  self-direction  has  been  under  a constant  iteration  process.  This  research  study  is 
primarily  focused  on  the  resulting  output  of  the  model.  The  question  and  research  guiding  the  current  research 
study  is  as  follows:  What  are  the  self-perceived  learning  gains  of  students  engaged  in  a social,  self-directed 
learning  experience?  A sub-question  to  this  is:  Do  students  improve  their  learning  in  objectives  that  are  not 
specifically  selected  on  self-designed  individual/group  learning  contracts?  The  results  of  this  research  will 
impact  the  overall  validity  of  the  model  and  provide  fodder  for  future  research  and  model  iteration. 

The  Instructional  Context 

The  instructional  intervention  that  was  applied  as  part  of  this  research  project  has  been  utilized  in  both 
web-based  (75%  or  more  time  spent  online)  and  hybrid  course  formats  (a  mixture  of  face-to-face  and  online 
experiences  making  up  either  a 40% -60%  or  50% -50%  split  of  time  in  either  setting).  Technology  integration 
courses  at  both  the  Masters  and  Undergraduate  level  have  been  utilized  for  the  described  intervention.  Student 
technology  experience  has  quite  varied  ranging  from  beginner  to  expert  levels. 

As  part  of  the  course  under  investigation,  students  participated  in  a one-day  face-to-face  orientation  in 
which  the  following  elements  were  introduced:  courseware  program  (Blackboard  6.0),  learning  patterns 
assessment,  self-diagnostic  instrument  completed  (pre-test),  course  material  reviewed,  expectations  shared, 
groups  formed,  and  learning  contracts  concepts  presented.  Rough  drafts  of  the  group  learning  contracts  were 
developed  prior  to  leaving  the  first  class  meeting. 

Individual  and  group  learning  contracts  were  created  based  on  the  areas  of  learning  “need”  self- 
determined  on  the  initial  diagnostic  instrument,  which  is  patterned  from  Knowles  (1986)  needs  assessment 
design.  Next,  the  following  components  were  identified  during  the  learning  contract  process:  strategies  and 
resources  that  would  be  used  to  complete  their  work,  dates  for  completion,  evidential  products  that  will 
demonstrate  new  knowledge,  and  authentication  procedures  to  verify  product  content  and  quality.  The  learning 
contract  process  is  graphically  portrayed  in  Figure  2. 

Students  were  also  required  to  remain  engaged  through  participation  in  the  online  discussion  board 
activities  and  completion  of  student  homepages.  Weekly  resources  (offline  content  and  online  materials)  were 
shared  throughout  the  semester  to  provide  students  with  a breadth  of  exposure  to  add  to  the  depth  of  objective 
exploration  that  occurs  as  a process  of  the  learning  contract  design.  The  diagnostic  instrument,  learning 
contract,  course  materials,  and  reflective  instruments  were  utilized  as  scaffolds  to  structure  and  guide  the 
experience  despite  the  previous  level  of  technological  knowledge. 

In  order  to  facilitate  the  development  and  sustenance  of  the  learning  communities,  which  is  primary  to 
the  model,  students  are  asked  to  create  at  least  three  group  objectives  on  the  learning  contract  and  another  two 
individual  objectives.  Students  can  decide  to  complete  all  group  objectives,  but  not  all  individual  objectives. 

The  use  of  group  objective  building  process  aims  at  reducing  isolation,  building  community  constructs, 
establishing  team/community  and  personal  learning,  developing  shared  vision,  and  creating  knowledge 
construction  as  has  been  deemed  vital  throughout  the  literature  on  learning  communities  and  communities  of 
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practice  (Brown,  2001;  Derry  & DuRussel,  2000;  Tu  & Correy,  2002).  Also  evident  in  the  proposed  model  and 
designed  environment  are  elements  of  Wenger’s  (2000)  modes  of  belonging;  engagement,  imagination,  and 
alignment. 


Methods 

A design-based  research  methodology  has  been  used  for  the  overall  model  development  to  focus  on  the 
global  perspective  of  this  unique  system  of  learning.  Design  based  research  has  been  found  to  be  appropriate  for 
model  design  and  iteration  in  complex  learning  environments  (Cobb,  Confrey,  diSessa,  Lehrer,  & Schauble, 
2003;  Sloane  & Gorard,  2003).  The  design-based  research  methods  provide  a means  for  macro  analysis  that  is 
necessary  to  bridge  multiple  concepts  in  complex  learning  environments. 

This  particular  phase  of  study  utilized  a non-experimental  research  design  with  a pre  -post  assessment 
of  self-perceived  level  of  learning  and  accomplishment  as  a result  of  the  instructional  design  intervention.  At 
the  time  of  presentation  there  were  eight  sets  of  data  with  an  additional  three  semesters  of  pilot  study.  The  data 
collection  involved  a pre  and  post  self-rated  diagnostic  instrument,  which  lists  all  of  the  course  objectives  and 
asks  students  to  rate  their  current  level  of  knowledge  (pre),  and  the  level  of  knowledge  necessary  to  be 
successful  in  their  anticipated  profession  role  (pre -relevancy  indicator).  The  output  measure  (post)  self-rated 
diagnostic  instrument  was  patterned  exactly  as  the  initial  pre -assessment  with  the  same  competencies  listed; 
however,  the  students  are  asked  to  rate  their  level  of  accomplishment  as  a result  of  the  instructional  experience. 
Students  are  not  “graded”  on  this  and  submission  of  this  instrument  in  no  way  affects  course  grades. 

Learning  contract  portfolio  documentation  and  a final  course  updates  were  used  to  validate  the  data 
gained  from  the  self-rated  diagnostic  instruments.  As  part  of  the  learning  process  students  designed  individual 
and  group  learning  contracts  and  resulting  products  and  authentication  that  show  evidence  of  competence 
acquisition.  These  portfolios  were  reviewed  using  document  analysis  to  further  triangulate  the  post-test 
instrument  data. 

The  final  course  update  includes  a final  reflection  posted  on  the  asynchronous  discussion  board  asking 
students  to  review  their  learning  over  the  semester  and  to  critically  examine  the  experience. 

Population 

Two  different  sample  groups  were  used  in  this  investigation,  but  all  would  be  considered  adult  learners 
ranging  from  age  from  25-65.  There  were  87  females  and  25  males  in  the  sample.  The  sample  group  was 
compiled  over  eight  separate  applications  of  the  course  intervention  with  students  over  a period  of  18  months 
and  three  different  course  titles.  There  was  great  diversity  in  student  technology  competence  (course  content 
area)  at  the  outset  of  each  section.  Appropriate  university  approvals  and  subject  consents  were  garnered  for  this 
research  study,  with  students  who  preferred  not  to  participate  being  removed  from  the  sample. 

The  sample  groups  all  utilized  the  same  instructional  strategies  and  formats  with  some  distinction  on 
final  expectation,  adapting  to  align  with  actual  course  objectives.  The  student  participants’  professional 
backgrounds  were  diverse  with  the  majority  of  students  coming  from  educational  fields;  however,  some  were 
from  nursing,  business,  pharmacology,  and  information  systems. 

Results 

Each  instrument,  pre  and  post,  were  analyzed  separately  for  descriptive  statistics.  Student  scores  for 
the  pre  self-rating  instrument  were  compiled  across  semesters  with  mean  values  run  by  question  on  the  pre  and 
post  instruments.  The  pre  and  post  assessment  included  24  self-rated  questions,  in  which  students  ranked  their 
knowledge  on  the  associated  course  objectives  on  a scale  from  1 (no  current  knowledge)  to  4 (high  current 
knowledge).  The  pre-test  question  means  ranged  from  1.53  to  3.13.  The  n value  for  each  question  varied 
dependent  upon  student  response  and/or  course  alignment  of  questions  (112-90).  There  was  significant 
variance  in  the  questions  with  answers  ranging  from  a minimum  rating  of  1 and  a maximum  rating  of  4 for  most 
questions,  which  supports  the  starting  diversity  of  technology  levels.  See  Table  1 for  a listing  of  the  descriptive 
statistics  by  question. 

The  post  assessment  followed  the  same  configuration  as  the  pre  assessment.  Students  rated  the  gained 
knowledge  on  the  course  objectives.  Means  ranged  from  2.67  to  3.7 1 with  an  n for  each  question  ranging  from 
90-112.  There  was  less  variance  in  the  post  scores,  but  a range  of  responses  were  still  evident.  A quarter  of  the 
questions  received  a minimum  of  a 2 rating  and  all  questions  received  maximum  ratings  of  4.  Table  1 provides  a 
listing  of  the  descriptive  statistics  for  the  post  assessment.  Student  responses  on  gained  knowledge  were  in  no 
way  used  to  establish  grades  and  were  submitted  electronically  with  no  “grade”  assigned  to  this  task. 
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The  pre  and  post  data  were  analyzed  using  a t-test  on  the  question  means  of  the  pre  -post  differences  to 
determine  the  significance  of  the  differences.  All  questions  demonstrated  significant  differences  (pc. 001) 
indicating  that  globally  learning  occurred  on  all  course  items.  The  t-scores  range  from  7.305  to  15.747.  Table  2 
provides  information  on  the  t-scores  by  question  number  with  the  respective  degrees  of  freedom. 

Students  received  feedback  on  the  learning  contract  objectives  they  submitted  as  their  evidence  for 
learning  that  was  predetermined  at  the  beginning  of  the  semester.  Students  were  given  detailed,  ejtensive 
feedback  on  their  initial  submission  based  on  a skill  rubric  that  was  shared  at  the  outset  of  the  semester  and 
were  then  provided  with  the  opportunity  to  improve  product  quality.  Through  this  process,  the  self-designed 
authentic  products  satisfactorily 

Table  1 .Descriptive  Statistics  on  Pre-Test  and  Post-Test  by  Question 

Pre  Post 


Question 

N 

Mean 

Standard 

Deviation 

Variance 

Skewness 

N 

Mean 

Standard 

Deviation 

Variance 

Skewness 

A1 

112 

3.13 

.65 

.42 

-.33 

112 

3.71 

.47 

.22 

-1.22 

A2 

112 

2.14 

.85 

.72 

.35 

112 

3.2 

.76 

.57 

-.60 

A3 

92 

2.01 

.78 

.60 

.55 

92 

2.93 

.81 

.66 

-.52 

A4 

112 

2.38 

.92 

.85 

.07 

111 

3.64 

.61 

.38 

-1.99 

A5 

110 

2.17 

.89 

.79 

.38 

110 

3.22 

.82 

.67 

-.84 

A6 

91 

1.79 

.80 

.63 

.67 

91 

2.86 

.91 

.83 

-.42 

B 1 

92 

1.73 

.83 

.68 

.91 

92 

2.98 

.78 

.61 

-.52 

B2 

92 

1.85 

.81 

.66 

.54 

92 

3.12 

.78 

.61 

-.78 

B3 

91 

2.16 

.87 

.76 

.29 

92 

3.34 

.72 

.51 

-.79 

B4 

91 

2.14 

.80 

.63 

-.13 

91 

3.36 

.66 

.43 

-.55 

Cl 

112 

2.95 

.79 

.63 

-.46 

112 

3.80 

.42 

.18 

-1.91 

C2 

112 

2.76 

.75 

.56 

.04 

112 

3.58 

.58 

.34 

-1.03 

C3 

92 

2.53 

.76 

.58 

-.04 

92 

3.64 

.55 

.30 

-1.21 

C4 

91 

2.66 

.72 

.52 

.06 

92 

3.58 

.56 

.31 

-.89 

C5 

112 

2.55 

.87 

.75 

.08 

112 

3.58 

.67 

.44 

-1.70 

D1 

92 

1.53 

.70 

.49 

.95 

92 

3.20 

.73 

.53 

-.67 

D2 

92 

1.71 

.76 

.58 

1.01 

92 

3.10 

.77 

.59 

-.46 

D3 

92 

1.88 

.80 

.63 

.47 

92 

3.16 

.75 

.56 

-.44 

D4 

111 

2.65 

.88 

.78 

-.28 

112 

3.57 

.65 

.43 

-1.85 

El 

92 

2.89 

.87 

.76 

-.09 

92 

3.68 

.57 

.33 

-2.03 

E2 

92 

1.79 

.90 

.80 

.80 

92 

2.83 

.98 

.96 

-.50 

E3 

91 

1.57 

.90 

.80 

1.53 

90 

2.67 

.92 

.85 

-.16 

E4 

91 

2.70 

.92 

.86 

-.14 

91 

3.57 

.63 

.40 

-1.47 

E5 

90 

2.61 

.87 

.76 

.12 

91 

3.58 

.67 

.45 

-1.57 
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Table  2.  T-Test  Scores  by  Question 


Question 

T Statistic 

Df 

Number  of  students  who 
selected  this  objective 

A1 

7.792 

222 

6 

A2 

9.810 

222 

59 

A3 

7.897 

182 

15 

A4 

11.948 

221 

76 

A5 

9.093 

218 

43 

A6 

8.391 

180 

30 

B 1 

10.522 

182 

24 

B2 

10.825 

182 

16 

B3 

9.943 

181 

11 

B4 

11.256 

180 

8 

Cl 

10.110 

222 

30 

C2 

9.169 

222 

16 

C3 

11.337 

182 

12 

C4 

9.639 

181 

8 

C5 

9.929 

222 

28 

D1 

15.747 

182 

41 

D2 

12.296 

182 

11 

D3 

11.280 

182 

31 

D4 

8.891 

221 

16 

El 

7.305 

182 

29 

E2 

7.463 

182 

15 

E3 

8.094 

179 

11 

E4 

7.382 

180 

8 

E5 

8.430 

179 

9 

**  All  questions  exhibit  significance  at  a p>.0001 
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met  the  designed  objectives  and  demonstrated  gained  knowledge.  Given  that  students  had  the  option  of  actively 
improving  their  work  based  upon  rubric  feedback,  few  assignments  across  the  multiple  semesters  received 
unsatisfactory  evaluations.  The  significant  results  from  the  t-test  pre-post  means  by  question  were  validated 
through  the  document  analysis  of  previous  work  products  and  instructor  ratings. 

Significance  of  achieved  breadth  of  knowledge  having  been  found,  it  was  then  necessary  to  investigate 
whether  the  selected  objectives  placed  on  the  learning  contract  by  students  resulted  in  higher  degree  of  learning. 
Growth  was  seen  across  all  questions  indicating  diffused  overall  learning  despite  student  concentration  on 
selected  objectives.  See  Table  2 for  a breakdown  of  the  number  of  students  who  selected  each  objective  for  the 
“depth”  portion  of  the  content.  Not  all  students  received  perfect  scores  or  a high  level  of  learning  based  upon 
the  objectives  that  were  selected.  However,  the  average  growth  on  those  objectives  selected  was  1.36,  with  a 
range  of  responses  between  a loss  of  a point  to  3 points  growth.  The  average  growth  on  those  objectives  not 
selected  was  .94,  with  the  range  of  responses  including  a loss  of  a point  to  3 points  of  growth.  While  it  would 
appear  unusual  for  students  to  have  selected  a value  indicative  of  a reduction  in  amount  of  knowledge  gained, 
this  does  perhaps  suggest  that  students  might  have  misjudged  their  initial  knowledge  due  to  lack  of 
understanding  of  terminology  and  basic  technology  skill.  These  data  definitely  suggest  that  perhaps  students 
were  not  concerned  about  instructor  “acceptance”  and  did  indeed  truly  self-evaluate  knowledge  at  both  points  in 
time. 


Conclusion 

The  question  of  “Who’s  in  charge?”  was  not  posed  in  an  attempt  to  remove  the  instructor  role  from  the 
“dance”  of  learning;  rather,  the  focus  was  on  investigating  whether  learning  did  occur  when  the  control  of 
learning  tasks  were  transferred  to  the  students  and  student  communities.  Instructors  play  a critical  role,  within 
the  model  and  instructional  design  of  social,  self-direction,  via  instructional  design,  student  facilitation,  content 
development,  feedback/critical  analysis,  and  evaluation.  However,  rather  than  the  instructor  guiding  the 
process,  the  learner  becomes  the  one  guiding  the  “dance”  direction,  flow,  meaning,  and  pace. 

The  model,  which  has  been  represented  in  the  formula  (l+P+0=0u ) is  shown  in  Figure  1,  and 
attempts  to  merge  popular  adult  education,  social  learning,  and  constructivist  philosophies  and  provide  a 
systemic  view  of  the  learning  process  that  incorporates  what  is  known  about  higher  education  best  practices  into 
the  online  environment.  Like  all  new  models,  further  iteration,  development  and  dimensional  visualization  will 
need  to  occur  to  generalize  and  replicate  the  system  elements.  However,  this  particular  study  was  focused  on 
understanding  whether  learning  was  occurring  given  the  instructional  intervention  that  was  attempted  in  the 
online  environment. 

The  data  of  the  pre  and  post  instruments  indicate  that  indeed,  students  self-rated  their  knowledge  gain 
as  significant.  All  questions  indicated  s ignificant  learning  of  the  objectives  (pc. 001),  which  is  substantiated  by 
the  learning  contract  products.  The  gradient  range  of  improvement  for  each  of  the  question  (from  a 1-pre  value 
to  a 3-post  value  indicator  OR  from  a 2-pre  value  to  a 4 -post  value  indicator)  was  different  for  each  individual 
based  upon  the  beginning  level  of  need,  the  objectives  that  might  have  received  greater  “depth”,  and  individual 
student  mediating  variables  (such  as  time,  available  technology,  etc.).  It  was  possible  that  the  high  level  of 
significance  that  was  found  included  some  “approval”  seeking  behaviors  from  the  students  involved  in  the 
study.  . For  instance,  given  the  semesters  worth  of  work  and  the  knowledge  of  the  ongoing  research  study,  it  is 
possible  that  the  students  rated  themselves  higher  to  “please”  the  instructor.  This  effect  is  somewhat  diminished 
by  the  lack  of  feedback  and  encouragement  on  this  document.  Further,  given  that  some  individual  scores 
showed  negative  values  in  objective  growth  makes  this  phenomenon  somewhat  unlikely.  The  instructor 
introduced  the  post  test  as  a reflective  assignment  that  will  in  no  way  affect  grading  and  suggests  that  the 
instrument  simply  be  used  as  a tool  for  creating  a learning  plan  and  reflecting  on  gained  knowledge. 

Each  learning  contract  was  evaluated  by  the  instructor  on  pre-established  rubrics  and  was  show  to 
either  be  initially  acceptable  or  through  an  active  learning  process  improved  to  demonstrate  appropriate 
knowledge  and  skill  growth.  This  process  was  used  as  a validation  indicator  to  establish  if  the  learning  products 
supported  the  student  self -perceived  learning  gains.  Students  did  have  evidential  products  that  met  instructor 
expectation  in  the  areas  that  were  selected  as  learning  objectives  for  the  course.  This  substantiates  the  self- 
perceived  knowledge  on  only  those  areas  that  were  included  on  the  learning  contract  (depth  of  knowledge  in 
choice  areas)  and  cannot  be  used  as  evidence  to  validate  the  breadth  of  knowledge  that  was  rated  on  the  post 
instrument. 

While  depth  of  knowledge  gained  in  the  class  was  found  to  be  significant  for  the  areas  selected  on  the 
learning  contract,  further  analysis  was  necessary  to  determine  if  growth  occurred  across  all  areas  to  attend  to  the 
breadth  of  content  that  the  course  was  slated  to  cover.  The  data  indicate  that  there  was  higher  average  growth  in 
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those  areas  that  were  selected  on  the  learning  contract.  The  increased  knowledge  gain,  in  areas  where  depth  was 
obtained,  was  substantiated  by  student  comments  on  final  reflective  updates . 

The  findings  of  this  study  provide  support  for  the  significant  learning  gains  that  are  achieved  when 
using  this  instructional  intervention.  The  next  step  in  this  process  will  be  the  need  to  tie  these  learning  gains  to 
the  much  more  meaningful  dimension  of  outcomes  that  has  not  yet  been  visually  designed.  The  outcomes 
element  provides  a way  to  conceptualize  the  impact  of  the  individual  student  learning  on  others  both  internally 
and  externally.  Were  there  other  valuable  and  meaningful  learnings  (outside  of  course  content)  that  occurred  as 
a result  from  the  use  of  this  model  in  the  instructional  design?  Do  students  become  more  self-directed  as  a result 
of  the  process  in  this  instructional  design  and  model  of  social,  self-direction?  Are  students  better  able  to  assume 
responsibility  for  their  own  learning  and  generalize  this  to  other  learning  situations?  Does  the  social,  self- 
directed  model  increase  leadership  potential  and  enhance  leadership  characteristics? 

In  fact,  a study  has  recently  been  conducted  showing  that  this  model  increases  the  use  of  leadership 
characteristics  such  as  the  use  of  time  management  skills,  organization  skills,  self-motivation  skills,  problem 
solving,  and  team/group  facilitation  skills  (Boyer,  2004).  However,  little  is  known  about  whether  these  learned 
skills  continue  throughout  further  coursework  given  a return  to  traditionally  designed  courses,  or  whether 
students  return  to  the  “comfort”  of  becoming  compliant  learners.  Further,  longitudinal  work  is  needed  to 
determine  whether  participant  comments  about  changes  in  their  organizational  environments  and  families  due  to 
this  instructional  intervention  are  indeed  signs  of  significant  external  impact.  In  other  words,  does  the 
instructional  design  of  a social,  self-directed  environment  transform  not  only  the  enrolled  individuals,  but  those 
outside  of  the  course  who  are  touched  by  those  who  are  involved  in  the  intervention? 

The  results  of  the  this  study,  provide  additional  support  for  the  possibility  of  designing  student- 
centered,  community  driven,  self-directed,  and  meaningful  environments  in  online  settings  that  provide  the 
opportunity  for  significant  learning  gains.  Students  can  be  responsible  for  personal  growth  without  the 
mandates  and  structures  that  many  who  have  been  experimenting  with  online  delivery  have  determined  are 
necessary.  In  order  to  help  students  be  successful  in  the  more  self -regulating  environments  scaffolds  may  need 
to  be  provided  (depending  upon  their  current  level  of  development).  Providing  these  scaffolds  adheres  to 
positive  adult  learning  practice  and  is  much  more  conducive  and  transforming  than  returning  to  the  “comforts” 
and  “traditions”  of  instructor  -led,  instructor-controlled  delivery  and  design. 

The  dance  of  learning  requires  a couple,  a partnership,  a community,  that  can  move  together  to 
transform  perspectives  and  enhance  knowledge  acquisition.  Providing  students  with  the  “lead”  opportunity  is 
oftentimes  not  appreciated  and/or  welcomed  given  previous  training  and  enculturation.  The  model  of  social, 
self-directed  learning  is  one  that  required  full  engagement,  participation,  and  commitment.  Some  students 
express  a longing  for  the  “ease”  of  sitting  in  class  for  a number  of  hours  being  told  what  to  do  and  how  to  do  it. 
This  perspective  limits  the  personal  investment,  which  has  been  necessary  to  be  functional  citizens  in  a 
knowledge  society  where  the  need  to  think,  plan,  learn,  collaborate,  and  innovate  is  paramount.  The  linearity  of 
the  “traditional”  process  was  much  “easier  and  simpler”;  however,  the  comp  lexity  and  commitment  that  is 
derived  from  a systems  model  such  as  the  one  presented  has  the  potential  of  altering  perspectives,  instilling 
personal  freedom  and  responsibility,  and  extending  the  learning  process  to  internal  and  external  communities. 
The  systems  model  of  social,  self-direction  aims  at  meeting  this  transitional  need  as  we  transform  from  a 
populace  of  compliant  thinkers  to  innovative,  self-motivated,  community  members. 
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Figure  1.  Model  of  Social,  Self-direction  for  online  environments  that  includes  input,  process,  output  and 
feedback  dimensions. 
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Figure  2.  A Process  for  developing  self-directed  learning  contracts  to  facilitate  personal  control  of  learning. 
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Abstract 

Two  types  of  technology  integration  have  been  the  subject  of  much  discussion  and  research  over  the 
past  several  years.  These  two  types  involve  the  integration  of  technology  into  the  classroom  and  the  integration 
of  diverse  learners  into  the  classroom.  Technology  integration  can  take  many  forms  ranging  from  simple 
remedial  programs  to  programs  and  systems  that  promote  creative  thinking  in  the  classroom.  Diversity  can 
also  be  complex  in  that  it  can  include  ELL  students,  students  with  disabilities,  and  issues  of  gender.  This  article 
will  explore  ways  in  which  technologies  that  enhance  teaching  and  promote  critical  thinking  can  be  integrated 
into  the  diverse  classroom  using  an  instructional  technique  known  as  differentiated  classroom  instruction. 

Type  II  Technology  Applications  and  the  Diverse  Learner: 

Issues,  Instruction,  and  Individuality 

Two  types  of  integration  have  been  the  subject  of  much  discussion  and  research  over  the  past  several 
years.  These  two  types  involve  the  integration  of  technology  into  the  classroom  and  the  integration  of  diverse 
learners  into  the  classroom.  Technology  integration  can  take  many  forms  ranging  from  simple  remedial 
software  to  software  and  systems  that  promote  creative  thought  in  the  classroom.  Software  and  systems  that 
promote  creativity  and  intellectual  growth  among  students  by  allowing  the  teacher  to  teach  in  new  and  better 
ways  are  commonly  known  as  Level  II  applications  (Maddux,  2002).  Integrating  diverse  learners  in  the 
classroom  can  also  take  many  forms.  The  process  of  integrating  students  with  disabilities  in  the  classroom  is 
commonly  known  as  inclusion. 

While  the  integration  of  technology  in  the  classroom  can  be  difficult,  it  can  help  teachers  meet  many  of 
the  challenges  presented  in  an  inclusive  classroom.  Although  many  technological  applications  for  students  with 
disabilities  tend  to  focus  on  remediation,  this  article  focuses  on  those  applications  and  situations  that  move 
away  from  remediation  and  help  stimulate  students’  intellectual  abilities  and  enhance  teaching  - that  is,  Level  II 
applications.  In  addition,  this  article  describes  these  applications  and  the  instructional  techniques  that  the 
classroom  teacher  can  use  to  facilitate  the  integration  of  technology  into  an  inclusive  classroom,  offering 
illustrative  examples  based  on  a combination  of  the  authors’  K-12  teaching  experiences  and  observations. 
Discussions  of  key  concepts  and  issues  such  as  differentiated  instruction  and  the  inclusive  classroom  are  also 
included. 


Differentiated  Instruction:  What  Is  It  And  How  Does  It  Work? 

Differentiated  Instruction  is  a well-researched  and  effective  instructional  technique  that  can  help  the 
classroom  teacher  meet  the  unique  needs  of  diverse  students  (Tomlinson,  2001).  Differentiated  instruction 
differs  from  individualized  instruction,  because  it  provides  learners  with  more  options  for  learning  while 
ensuring  that  the  classroom  workload  remains  manageable  for  the  teacher.  A teacher  creates  this  balance  by 
looking  at  similarities,  as  well  as  differences,  among  the  students  in  his  or  her  classroom.  By  grouping  students 
together  to  handle  the  unique  instructional  needs  of  each  group,  the  teacher  can  strike  a balance  between 
meeting  the  needs  of  a variety  of  learners  with  different  abilities  and  experiences  with  the  reality  of  the 
classroom  where  one  teacher  is  expected  to  work  with  as  many  as  30  students.  Another  advantage  of 
differentiated  instruction  is  that  it  can  effectively  utilize  a wide  variety  of  assistive/adaptive  technologies. 
Assistive/adaptive  technologies  are  those  that  help  a person  with  common  tasks,  making  something  physically 
accessible  that  would  be  inaccessible  otherwise;  wheelchairs,  Braille  readers,  and  hearing  aids  are  examples  of 
assistive/adaptive  technologies. 
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Differentiated  instruction  is  uniquely  suited  to  help  teachers  handle  the  unique  learning  characteristics 
of  a wide  variety  of  students  by  allowing  teachers  to  more  effectively  integrate  limited  technology  resources 
into  the  day-to-day  activities  of  the  classroom.  In  order  for  technology  to  be  most  effective,  technology  must  be 
fully  integrated  into  the  classroom.  Because  very  few  teachers  have  a classroom  full  of  computers  or  a laptop 
for  every  student,  an  instructional  technique  that  both  meets  the  unique  instructional  needs  of  each  student  while 
providing  accessibility  to  the  available  technology  is  important.  Differentiated  instruction  does  this  well. 

Differentiated  instruction  can  also  be  thought  of  as  a way  of  teaching  that  provides  students  with  many 
ways  to  access,  process,  and  output  information  that  best  meets  his  or  her  instructional  needs.  In  a 
differentiated  classroom,  the  teacher  assesses  the  students’  instructional  needs,  creates  and/or  uses  instructional 
techniques  that  meet  those  needs;  the  teacher  then  reassesses  the  students’  needs  to  inform  the  next  round  of 
instruction.  Carol  Tomlinson  (2001)  describes  the  key  characteristics  that  form  the  core  of  differentiated 
classroom.  The  first  of  these  is  that  the  instruction  is  proactive;  the  teacher  is  constantly  anticipating  the 
instructional  needs  of  both  the  individual  student  and  the  group.  Another  characteristic  is  that  the  instruction  is 
qualitative;  the  teacher  does  not  just  assign  more  or  less  work  but  tailors  the  work  to  the  student's  unique 
instructional  needs.  A third  characteristic  of  differentiated  instruction  is  the  use  of  formative  assessment;  the 
teacher  uses  the  assessments  given  in  the  classroom  to  continually  modify  instruction  for  the  student.  In 
differentiated  instruction  the  teacher  takes  multiple  approaches  to  the  content  presented,  the  instructional 
process,  and  student  products;  the  goal  of  the  teacher  is  to  try  and  find  the  best  match  between  the  student  and 
the  instruction. 

Differentiated  instruction  is  also  student  centered,  flexible  and  engages  the  student  at  a level 
appropriate  for  that  student.  What  makes  differentiated  instruction  different  from  individualized  instruction  is 
that  it  is  a mix  of  whole -class,  small  group,  and  one-on-one  instruction.  This  is  important  because  it  allows  the 
teacher  to  effectively  manage  the  classroom  by  addressing  the  needs  of  four  or  five  groups  rather  than  thirty 
individual  students. 

Technology  and  Differentiated  Instruction:  A Case  Study 

To  see  how  differentiated  instruction  might  work  in  a classroom,  we  will  use  the  example  of  Mrs. 
Winslow’s  classroom.  Although  Mrs.  Winslow’s  classroom  is  a fictional  construct,  the  premises  are  based 
upon  the  authors’  experiences  in  and  observations  of  actual  elementary  classroom  populations  and  settings. 

Mrs.  Winslow  has  an  inclusive  classroom  of  elementary  school  students.  During  a particular  instructional  unit, 
all  of  her  students  are  reading  a popular  book.  She  has  organized  her  students  into  three  groups.  One  group  of 
students  has  been  identified  as  dyslexic.  These  students  have  no  problem  comprehending  the  subject  matter  but 
have  difficulty  decoding  the  text  in  the  book.  The  second  group  consists  of  students  in  the  regular  education 
program.  These  students  have  average  reading  and  comprehension  abilities.  The  third  group  is  a group  of 
academically  gifted  students  who  have  already  read  the  book  at  least  three  times. 

Normally,  Mrs.  Winslow  has  all  of  her  students  complete  a traditional  book  report  in  which  each 
student  has  to  write  a book  report  that  describes  the  story  of  the  book  and  asks  the  students  to  draw  their  own 
conclusions  about  the  book.  In  order  to  better  meet  the  needs  of  her  students,  Mrs.  Winslow  decides  to 
differentiate  the  assignment.  She  begins  by  assessing  the  students'  technology  competencies,  asking  them  about 
their  familiarity  with  programs  such  as  word  processing,  Web  browsing,  and  Web  page  creation.  She  finds  that 
many  of  the  students  identified  as  academically  gifted  have  at  least  some  experience  with  Web  page  production, 
a fair  amount  of  experience  with  word  processing,  and  a great  deal  of  experience  with  Web  browsing.  The 
students  identified  as  dyslexic  have  a wide  variety  of  technical  skills  from  no  skills  with  the  computer  to  being 
able  to  develop  Web  pages.  The  general  education  students  also  have  a great  deal  of  experience  with  computers 
but  their  experiences  have  focused  on  "Web  surfing"  and  some  word  processing. 

Mrs.  Winslow  decides  to  differentiate  her  instruction  at  two  levels  based  on  this  pre -assessment.  She 
begins  by  having  all  of  the  students  read  the  book,  but  in  different  ways.  The  students  who  have  been  identified 
as  dyslexic  listen  to  an  audiotape  copy  of  the  book  that  Mrs.  Winslow  has  obtained  from  the  Library  for  the 
Blind.  These  students  listen  in  pairs  or  small  groups  with  multiple  headphones  plugged  into  the  tape  player 
using  equipment  also  provided  by  the  Library  for  the  Blind.  If  one  student  does  not  understand  the  text  or  gets 
lost,  a group  leader  is  assigned  to  stop  and  rewind  the  tape  in  order  to  review. 

The  regular  education  students  are  grouped  together  and  read  the  book  to  each  other  in  pairs  or  small 
groups  of  three  to  four  students.  They  are  grouped  based  on  their  individual  reading  abilities  so  that  the 
stronger  readers  are  paired  or  grouped  with  the  weaker  students.  The  students  identified  as  academically  gifted 
are  asked  to  read  the  book  independently  with  the  understanding  that  they  may  ask  the  teacher  for  help  at  any 
time. 
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Once  all  of  the  students  have  finished  the  book,  everyone  works  on  writing  a book  report  in  pairs  or 
small  groups.  The  groups  ultimately  present  their  book  reports  to  the  class.  After  the  presentations,  Mrs. 
Winslow  guides  the  class  though  an  activity  in  which  the  best  parts  of  each  of  the  book  reports  are  collected  and 
organized  to  make  a "super"  book  report. 

Mrs.  Winslow  then  reorders  the  students  by  level  of  technology  skill.  The  students  with  high 
technology  skills  are  challenged  to  learn  new  Web  authoring  software  that  has  been  obtained  from  the  district 
office.  These  students  work  together  on  a computer  in  the  classroom  to  learn  the  technology  tool.  Meanwhile, 
the  students  with  weak  technology  skills  are  asked  to  begin  storyboarding  the  class  project.  A storyboard  is  a 
document  that  contains  the  report’s  text,  descriptions  of  pictures  for  the  report,  and  a graphical  representation  of 
how  the  content  for  the  Web  site  will  be  organized.  The  students  with  moderate  technology  skills  are  asked  to 
type  the  book  report  so  that  it  is  digitally  captured  in  the  computer.  They  are  also  responsible  for  gathering 
multimedia  elements  such  as  pictures  and  links  to  areas  of  interest  on  the  Web. 

Once  the  storyboard  is  completed,  Mrs.  Winslow  then  pairs  each  of  the  students  with  strong 
technology  skills  with  groups  of  students  with  moderate  to  weak  technology  skills.  The  storyboard  is  also 
divided  up  so  that  each  group  has  its  own  part  of  the  storyboard.  The  students  with  advanced  technology  skills 
then  help  the  students  with  weak  technology  skills  create  their  own  part  of  the  class  Web  site.  Because  the  Web 
site  can  be  run  from  the  computer's  hard  drive,  each  of  the  groups  take  turns  working  on  the  classroom 
computer.  Each  group  presents  its  portion  of  the  site  to  the  rest  of  the  class,  once  the  entire  project  is  completed. 

Sensitivity  to  Gender  and  Culture  While  Using  Technology  in  the  Classroom 

As  the  use  of  technology  in  the  classroom  to  address  the  needs  of  diverse  learners  continues  to 
increase,  many  issues  are  being  uncovered.  One  issue  that  has  generated  and  continues  to  generate  a significant 
amount  of  emphasis  within  education  and  the  press  is  the  use,  or  more  appropriately  the  lack  of  use,  of 
technology  by  minorities,  students  from  low  socio-economic  backgrounds,  and  females.  Awareness  of  this  issue 
will  help  teachers  be  sensitive  to  what  needs  to  take  place  for  these  students  to  benefit  from  the  use  of 
technology  as  it  is  integrated  into  classroom  instruction. 

The  term  that  is  commonly  used  to  describe  this  issue  is  the  digital  divide.  The  digital  divide  indicates 
the  gap  that  exists  between  those  who  have  access  to  technology  and  those  who  do  not.  Despite  the  increased 
availability  and  use  of  technology  in  the  classroom,  a divide  still  exists.  This  divide  assists  in  bringing  about  a 
serious  disconnect  between  those  who  have  access  to  and  training  in  the  use  of  technology  and  those  who  do  not 
(Pearson,  2001;  Hoffman  & Novak,  1998).  Those  who  suffer  most  frequently  from  the  effects  of  the  digital 
divide  are  minorities  (Bolt  and  Crawford,  2000).  Historically,  minorities  have  not  been  exposed  to  technology 
in  significant  ways,  often  because  technology  has  been  viewed  as  being  controlling  rather  than  empowering. 

This  perception  is  changing  as  minorities  embrace  the  benefits  that  technology  can  provide. 

Teachers  need  to  make  a conscious  effort  to  help  insure  that  minorities  have  access  to  technology. 
Opportunities  should  be  deliberately  created  and  provided  that  allow  them  to  have  as  much  time  as  possible  to 
use  technology  in  the  classroom.  Teachers  should  also  model  the  use  of  technology  to  show  that  having  skill  in 
using  technology  can  be  empowering.  By  doing  this,  many  of  the  fears  that  minorities  have  about  technology 
are  broken  down;  technology  no  longer  is  perceived  as  a threat  and  it  then  becomes  an  empowering  tool.  One 
way  in  which  this  can  occur  is  by  differentiating  instruction  in  a way  that  allows  students  to  explore  topics  that 
are  relevant  to  their  lives. 

Technology  and  the  ELL  Student 

English  language  learners  (ELL)  are  students  whose  primary  language  is  not  English.  ELL  students 
often  have  difficulties  mastering  English,  which  creates  several  barriers  for  them  in  mastering  content.  Along 
with  the  challenges  students  may  face  with  English  literacy,  students  may  well  be  illiterate  in  their  primary 
language.  This  increases  the  challenges  faced  by  teachers,  because  not  only  does  the  student  lack  English  skills, 
the  student  may  also  lack  the  basic  skills  needed  to  read  at  all.  The  ELL  student  may  have  a learning  disability 
that  can  interfere  with  learning  these  basic  skills.  At  the  other  extreme  is  the  ELL  student  who  is  gifted.  These 
students  pose  unique  challenges  for  the  classroom  teacher  who  must  not  only  help  the  student  acquire  English 
language  skills,  but  must  also  challenge  them  in  their  native  language.  A good  starting  point  to  help  students 
deal  with  these  issues  is  for  the  teacher  to  have  a very  good  grasp  of  where  each  student  is  in  terms  of  his  or  her 
skill  development.  Once  this  is  understood  then  ELL  students  can  be  grouped  appropriately  to  meet  their 
unique  needs. 

Technology  can  provide  the  teacher  with  some  very  useful  tools  that  can  help  them  address  these 
needs.  Several  Web  sites  and  software  programs  exist  that  can  translate  English  into  other  languages  such  as 
Spanish,  German,  or  Portuguese.  Conversely,  the  same  Websites  or  software  can  be  used  to  translate  Spanish 
into  English.  ELL  students  who  are  academically  advanced  may  benefit  from  a mini  research  project  that 
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involves  accessing  English  only  Web  sites  to  find  about  a specific  topic.  If  they  have  problems  interpreting  the 
English  text  found  on  these  sites,  students  can  translate  the  text  using  the  translation  Web  sites  or  software. 

Once  they  have  completed  the  paper  in  English,  the  teacher  can  then  have  students  translate  the  text  back  into 
Spanish  to  see  if  the  translated  text  is  what  they  had  intended  to  write. 

English  language  difficulties  can  also  exacerbate  the  often-difficult  endeavor  of  fitting  into  the 
classroom  culture.  ELL  students  may  experience  isolation  and  frustration  from  the  lack  of  being  able  to 
communicate  effectively  with  the  teacher  and  classmates.  This  can  lead  to  classroom  management  and 
discipline  problems.  In  order  to  address  these  issues  the  teacher  should  look  for  occasions  to  provide  ELL 
students  with  positive  opportunities  where  they  can  feel  they  are  part  of  the  classroom  culture  without  denying 
their  own  culture. 

The  fact  that  ELL  students  tend  to  have  much  less  access  to  technology  than  do  their  peers  (Neuman, 
1994)  means  that  the  teacher  needs  to  carefully  plan  for  the  inclusion  of  ELL  students  in  classroom  instruction 
that  involves  technology.  For  ELL  students,  technology  needs  to  be  part  of  a learning  environment  that 
encourages  discovery  learning  and  connections  to  larger  communities  (including  their  own  native  community) 
(Ovando,  1998).  Another  instructional  principal  for  ELL  students  is  the  creation  of  an  active  and  engaging 
environment  (Liaw,  1997).  "Children  need  to  be  able  to  interact  with  each  other  so  that  learning  through 
communication  can  occur.  By  using  electronic  tools  such  as  email,  moos,  and  video  conferencing  the  computer 
can  act  as  a tool  to  increase  verbal  exchange"  (Ybarra  and  Green,  2003). 

Typical  use  of  technology  by  ELL  students  tends  to  be  through  drill  and  practice  software  where 
students  interact  with  English  in  an  isolated  manner.  Although  drill  and  practice  software  has  its  place,  it  should 
not  be  the  sole  manner  in  which  ELL  students  interact  with  technology.  With  the  wide  variety  of  technology 
tools  that  are  available,  teachers  have  many  options  for  creating  meaningful  instruction  for  the  ELL  student  that 
go  beyond  the  drill  and  practice  software. 

There  are  several  principles  that  can  help  guide  teachers  when  using  technology  to  facilitate  ELL 
instruction  (Butler-Pascoe  and  Wiburg,  2003;  Brown,  1993).  The  first  of  these  principals  is  that  effective  ELL 
instruction  uses  technologies  that  will  let  students  create  their  own  work.  This  can  be  as  simple  as  having 
students  create  word -processed  documents  that  incorporate  digital  pictures  to  video  and  audiotapes,  which 
illustrate  the  student's  cultural  heritage.  The  most  involved  examples  involve  the  creation  of  multimedia  Web 
sites  that  combine  digitized  video,  audio,  and  pictures  with  text  and  links  to  create  a rich  exploration  of  a 
particular  culture  or  language.  A second  principle  is  to  allow  ELL  students  to  capture  oral  records  of  their  work, 
which  can  be  done  using  very  inexpensive  tape  recorders.  Additionally,  most  computers  in  classrooms  have 
audio  capture  cards  that,  along  with  an  inexpensive  microphone,  can  also  be  used  to  capture  examples  of  ELL 
students  work;  this  work  can  then  be  embedded  in  PowerPoint  presentations  or  Web  sites. 

In  addition  to  these  two  principals,  teachers  should  also  take  advantage  of  having  students  work 
together  in  pairs  or  small  groups  to  write  collaboratively.  These  groups  can  be  a mix  of  ELL  and  non-ELL 
students.  By  grouping  ELL  and  non-ELL  students,  ELL  students  can  work  on  improving  their  English  language 
skills  though  immersion  and  reinforcement.  If  a writing  assignment  involves  some  aspect  of  the  ELL  student's 
culture,  then  the  student  can  also  become  an  "expert"  on  a topic,  which  can  help  to  improve  the  student’s  sense 
of  self-worth.  If  a video  camera  is  available  a videotaped  book  report  can  be  created.  The  student  can  then  go 
back  to  either  the  audio  or  videotape  to  review  the  report  not  only  for  content  but  also  for  their  mastery  of 
English. 

In  addition  to  technology  that  is  used  strictly  within  the  classroom,  the  teacher  should  also  use 
technology  to  facilitate  communication  (using  the  Internet)  with  students  from  other  places  in  the  World.  This  is 
perhaps  one  of  the  most  powerful  uses  of  Level  II  technology  tools.  By  establishing  connections  with 
classrooms  around  the  world,  students  can  communicate  with  others  who  not  only  speak  their  own  native 
language  but  who  can  also  reinforce  cultural  ties.  At  the  same  time  students  in  the  class  who  are  native  English 
speakers  can  get  to  know  the  ELL  student's  cultural  experience  better,  and  thus  have  a greater  appreciation  of 
these  ELL  students.  This  can  be  done  with  e-mail  pals  from  other  classrooms  (sometimes  referred  to  as  key- 
pals).  Many  Web  sites  exist  that  can  provide  the  teacher  with  an  international  directory  of  other  school  sites 
that  can  be  used  as  a starting  point  for  teachers  wishing  to  contact  other  teachers  interested  in  establishing 
international  e-mail  pals. 


Gender  Equity  and  the  Use  of  Technology 

The  classroom  teacher  should  also  be  aware  that  inequities  exist  in  how  often  and  in  what  ways 
females  use  technology  in  comparison  to  males  (Gilley,  2002).  Boys  typically  use  computers  as  toys  while  girls 
use  computers  to  accomplish  specific  tasks.  Boys  often  tend  to  become  "obsessed”  with  using  technology  while 
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girls  tend  to  be  more  occasional  users.  This  has  led  to  boys  being  encouraged  by  adults  to  use  technology 
(Margolis  and  Fisher,  2001).  In  spite  of  this,  research  indicates  that  girls  are  just  as  capable  as  boys  in 
effectively  using  technology  even  though  female  perceptions  indicate  that  they  believe  they  have  less 
experience  and  knowledge  than  males  in  using  technology  (Gilley,  2002;  Mathis,  2002).  In  order  to  address  the 
unique  needs  of  girls  with  respect  to  technology,  teachers  need  to  ensure  that  female  students  have  equal  access 
to  technology  and  encourage  them  to  use  it  in  a variety  of  meaningful  ways. 

Creating  deliberate  opportunities  for  girls  to  use  technology  in  a variety  of  situations  as  well  as 
providing  role  models  for  girls  can  go  along  way  in  helping  to  bridge  the  gender  gap  in  the  classroom.  How  this 
might  be  done  can  be  found  in  the  case  study  example  of  Ms.  Martin  who  uses  her  computer  lab  time  to 
introduce  new  concepts  such  as  scanning  to  the  entire  class.  After  her  presentations,  however,  the  few  scanners 
available  to  the  class  tended  to  be  dominated  by  the  boys.  To  fix  this  problem  she  decided  to  conduct  small 
group  instruction  where  she  would  make  sure  that  everyone  in  the  group  had  an  equal  opportunity  to  learn  how 
to  use  the  equipment;  thus  allowing  the  girls  in  the  class  to  build  confidence  when  using  the  equipment. 

Using  Technology  in  the  Content  Areas  to  Facilitate  Learning  among  Diverse  Learners 

Several  examples  of  how  technology  can  be  used  within  content  areas  to  help  meet  the  needs  of 
diverse  learners  are  provided  in  this  section.  Many  of  the  sample  activities  provided  can  be  used  in  various 
content  areas.  It  is  important  to  note  that  the  examples  provided  are  only  a representative  sample  of  the  almost 
endless  possibilities  that  exist. 

Art 

Art  is  very  well  suited  for  the  integration  of  technology.  Most  relatively  new  computers  are  well 
equipped  with  the  capabilities  of  allowing  students  to  develop  their  creativity  through  the  creation  of  digital 
artwork.  There  are  numerous  graphic  software  programs  available  that  students  can  use  to  create  and  edit  their 
own  artwork.  The  artwork  can  be  distributed  in  many  different  ways  - printed,  distributed  on  a CD-ROM  or 
DVD,  or  simply  viewed  on  a computer  or  television  screen. 

Students  with  physical  disabilities  can  have  difficulties  manipulating  a standard  computer  mouse. 

There  are  numerous  alternative  devices  that  can  be  used  instead  of  a standard  computer  mouse  - trackball, 
touch-screen,  and  graphic  tablet.  Each  of  these  devices  makes  it  easier  for  students  with  disabilities  to  use  the 
computer  to  develop  their  own  digital  artwork. 

Language  Arts 

Technology  integration  into  language  arts  can  be  extremely  beneficial  for  students  with  diverse  needs. 
Technology  allows  students  to  work  on  essential  reading  and  writing  skills,  and  to  express  themselves  in  ways 
beyond  writing.  The  use  of  digital  books  in  the  context  of  language  arts  is  a great  way  to  meet  the  needs  of 
diverse  learners . 

Mathematics 

Technology  can  be  exceptionally  useful  in  helping  students  work  on  a variety  of  math  skills  from  basic 
operations  (e.g.  addition  and  subtraction)  to  more  complex  math  skills  (e.g.  logical  thinking  needed  to  solve 
geometric  proofs  or  algebraic  equations).  Computing  tools  are  especially  useful  in  math  because  of  the  power 
that  computers  have  in  quickly  processing  numbers  which  means  that  students  are  free  to  explore  more  complex 
mathematical  concepts  as  well  as  "what  if"  scenarios  more  quickly  than  if  they  have  to  work  out  problems  with 
paper  and  pencil. 

Science 

For  years  computers  have  played  an  important  role  in  scientific  exploration.  Computers  are  able  to 
recreate  complex  models  of  physical  and  organic  systems.  In  the  classroom,  computers  can  meet  the  needs  of  a 
wide  variety  from  those  who  need  remediation  on  certain  scientific  concepts  to  those  who  need  enhanced 
instruction.  Recent  developments  in  three-dimensional  modeling  can  even  allow  students  new  perspectives 
through  the  graphical  representation  of  anything  from  an  atom  to  a galaxy.  Lego  MindStorms  is  another 
product  that  can  facilitate  higher  order  thinking  skills  in  the  area  of  science.  It  is  a robotic  development  system, 
which  provides  students  with  opportunities  to  create  their  own  robots  (and  other  devices). 

Social  Studies 

A major  goal  of  social  studies  education  is  to  teach  students  to  understand  what  it  means  to  live  in  a 
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democratic  society.  There  are  numerous  elements  to  this  process,  some  of  which  are  learning  to  be  a productive 
citizen  and  respecting  other  people  and  their  cultures  (as  well  as,  understanding  one’s  own  culture).  Technology 
can  be  a wonderful  source,  especially  the  Internet,  in  helping  students  connect  to  and  learn  from  others,  which 
can  be  beneficial  in  helping  reach  the  major  goals  of  social  studies  education.  The  goals  of  the  social  studies 
curriculum  provide  excellent  opportunities  for  including  ELL  students  such  as  connecting  with  students  from 
across  the  country  and  around  the  world.  Social  studies  activities  also  provide  many  opportunities  for  the  use 
and  creation  of  rich  multimedia  documents  that  can  illustrate  many  concepts  in  new  and  exciting  ways.  While 
most  students  will  never  be  able  to  travel  to  different  locations  around  the  world  virtual  expeditions  are  real- life 
travels,  typically  consisting  of  a team  of  individuals  who  explore  different  locations  around  the  world. 

Summary 

The  integration  of  various  technologies  into  the  classroom  can  provide  learners  with  unique 
opportunities  to  help  meet  their  diverse  needs.  The  strategy  of  differentiated  instruction  can  help  make  best  use 
of  technologies  to  support  learners  without  placing  an  overwhelming  burden  on  the  teacher.  Technology  can  be 
an  empowering  tool.  For  it  to  be  empowering,  however,  teachers  must  be  deliberate  and  thoughtful  in  how  it  is 
integrated  and  used  in  the  classroom.  This  is  not  only  true  for  students  with  a variety  of  impairments  but  is  also 
true  for  ELL  and  gifted  and  talented  students  as  well.  Issues  of  gender  and  technology  use  have  also  become 
important  to  consider  as  computing  becomes  more  pervasive  in  our  society  and  schools.  Of  course,  while 
technology  can  be  used  to  help  develop  solutions  to  these  problems,  it  is  ultimately  the  teacher  using  and 
facilitating  the  use  of  the  technology  that  can  enrich  and  empower  students. 

References 

Bolt,  D.  B.  & Crawford,  R.  A.  K.  (2000).  Digital  divide:  Computers  and  our  children 's  future.  New  York: 
TVBooks. 

Brown,  K.  (1993).  Balancing  the  tools  of  technology  with  our  own  humanity:  The  use  of  technology  in 
building  partnerships  and  communities.  In  J.V.  Tinagero  & A.F.  Ada  (Eds.),  The  power  of  two 
languages:  Literacy  and  biliteracy  for  Spanish-speaking  students  (pp.  178-198).  New  York: 

M ac  m ilia  n/ M c G raw-  Hill. 

Butler-Pascoe,  M.  E.  & Wiburg,  K.  M.  (2003).  Technology  and  teaching  English  language  learners.  Boston: 
Allyn  & Bacon. 

Gilley,  J (2002).  Gender  and  technology  awareness  training  in  pre  -service  teacher  education.  TechTrends,  46 
(6),  21-26. 

Liaw,  M.L.  (1997).  An  analysis  of  ESL  children's  verbal  interaction  during  computer  book  reading.  Computers 
in  the  Schools,  73(3/4),  55-73. 

Maddux,  C.  D.  (2002).  The  web  in  education:  A case  of  unrealized  potential.  Computers  in  the  Schools, 
79(1/2),  7-17. 

Margolis,  J.  and  Fisher,  A.  (2001).  Unlocking  the  clubhouse:  Women  in  computing.  Cambridge,  MA:  MIT 
Press. 

Mathis,  S,G.  (2002)  Improving  first-year  women  undergraduates'  perceptions  of  their  computer  skills. 
TechTrends,  46(6),  27-29. 

National  Center  for  Education  Statistics  (2000).  Internet  access  in  U.S.  public  schools  and  classrooms:  1994- 
99. Washington:  US  NCES.  [Retrieved  February  12,  2004 
http://nces.ed.gov/pubs2000/2000086.pdf  ] 

Neuman,  D.  (1994).  Technology  and  equity.  NABE  News,  18(1),  17-18,  32. 

Ovando,  C.  J.,  & Collier,  V.  P.  (1998).  Bilingual  and  ESL  classrooms  (2nd  Ed.).  Boston:  McGraw  Hill 
Pearson,  T.  (2001).  Falling  behind:  A technology  crisis  facing  minority  students.  TechTrends,  46(2),  15-20. 
Tomlinson,  C.  A.  (2001).  How  to  differentiate  instruction  in  mixed-ability  classrooms  (2nd  Ed.).  Alexandria, 
VA:  Association  for  Supervision  and  Curriculum  Development. 

Ybarra,  R,  & Green,  T.  (2003).  Using  technology  to  help  ESL/EFL  students  develop  language  skills.  The 
Internet  TESL  Journal,  6(3).  [Retrieved  February  12,  2004:  http://itesli.org/Articles/Ybarra- 
Technology.html] 


139 


Small  Group  Learning  in  an  Online  Asynchronous  Environment 

Susan  A.  Brewer 
James  D.  Klein 
Arizona  State  University 

Abstract 

This  article  reports  the  results  of  a study  conducted  to  examine  the  use  of  small  group  learning 
strategies  in  an  online  college  course.  The  study  examined  the  effect  of  four  types  of  positive  interdependence 
and  the  affiliation  motive  on  learning  and  attitude  in  an  asynchronous,  collaborative  learning  environment. 
Results  indicated  no  significant  differences  in  achievement  by  type  of  interdependence,  or  by  affiliation  motive. 
Correlation  analysis  revealed  a significant  positive  correlation,  indicating  that  participants  with  higher 
numbers  of  interactions  attained  higher  posttest  scores.  Participants  in  reward  groups  had  significantly  higher 
agreement  with  several  attitude  statements  that  reflected  benefit  from  working  with  others  and  being  able  to 
generate  better  ideas  in  groups.  Furthermore,  participants  in  all  three  types  of  structured  interdependence, 
compared  to  groups  with  no  interdependence,  had  significantly  higher  agreement  with  being  able  to  learn  more 
because  team  members  knew  it  was  their  job  to  contribute  to  the  group  work.  In  addition,  participants  with  high 
affiliation  motive  had  significantly  higher  agreement  with  several  attitude  statements.  Groups  with  no 
structured  interdependence  had  the  most  cognitive  interactions,  role  groups  had  the  most  group  processing,  and 
reward  groups  were  most  off  task.  Implications  for  integrating  small  groups  in  computer-mediated  learning 
environments  are  discussed. 

Cooperative  Learning  and  Affiliation  Motive 

Over  the  past  decade,  there  has  been  increased  interest  in  using  cooperative  methods  in  college 
classrooms.  During  the  1990s,  more  than  170  studies  were  conducted  to  examine  the  influence  of  cooperative 
learning  on  college  students  and  other  adult  learners.  According  to  Johnson,  Johnson,  and  Smith  (1998),  the 
results  of  this  research  are  impressive  when  the  magnitude  of  effect  sizes  in  favor  of  cooperative  over 
competitive  and  individual  methods  are  considered.  However,  other  researchers  have  found  less  success  with 
cooperative  learning  for  enhancing  college  student  performance  (Niehoff  & Mesch,  1991;  Cole  & Smith,  1993; 
Klein,  Erchul  & Pridemore,  1994;  Klein  & Schnackenberg,  2000.)  Regarding  student  motivation,  some 
research  on  cooperative  learning  suggests  that  students  working  in  groups  are  more  motivated  than  those  who 
work  alone,  but  performance  in  these  settings  is  influenced  by  one's  affiliation  motives  (Klein  & Pridemore, 
1992).  Cooper  (1995)  notes  that  even  though  the  components  of  cooperative  learning  have  been  well  defined, 
systematic  research  on  the  efficacy  of  each  is  still  quite  scarce,  particularly  for  college  populations. 

Distance  Learning  for  Adult  Reentry  Students 

More  than  190  traditional  institutions  are  offering  accelerated  degree  programs  with  evening,  weekend, 
and  distance  features  that  cater  to  working  adults  (Wlodkowski,  Mauldin,  & Gahn,  2001).  Accelerated  programs 
for  adults  use  far  more  active  and  collaborative  learning  processes  than  exist  in  traditional  programs  (Scott  & 
Conrad,  1991).  In  the  same  manner  that  small  groups  in  campus-based  classes  help  to  overcome  anonymity 
(McKinney  & Buxton,  1993),  small  groups  in  distance  education  classes  may  also  reduce  anonymity  and 
isolation.  Research  on  computer- mediated  collaborative  learning  indicates  that  it  can  be  can  be  as  effective 
(Johnston,  1996)  or  more  effective  (Hall,  1997;  Naidu  et  ah,  1999;  Uribe,  Klein,  & Sullivan,  2003)  than  face-to- 
face  collaboration. 

In  1999-2000,  eight  percent  of  all  undergraduates  participated  in  some  type  of  distance  education  with 
one-third  of  those  being  enrolled  in  entirely  distance  education  programs  (National  Center  for  Education 
Statistics,  2002).  The  prevalence  of  online  collaboration  in  higher  education  is  supported  by  the  increasing 
availability  of  technology-mediated  instruction  and  by  growing  business  and  industry  demands  for  working  in 
teams  (Ben-Jacob  & Levin,  1998;  Mclsaac  & Gunawardena,  1996).  In  light  of  the  mixed  findings  on  what  is 
motivating  and  constitutes  effective  instructional  strategies  for  adults,  research  is  needed  to  clarify  the 
effectiveness  of  small  group  learning  for  adults  in  an  asynchronous  environment.  The  research  questions 
addressed  by  this  study  were: 
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1.  What  is  the  effect  of  type  of  positive  interdependence  on  achievement,  attitude,  and  interaction 
behavior  for  adult  reentry  students  in  an  asynchronous  collaborative  learning  environment? 

2.  What  is  the  relationship  between  affiliation  motive  and  achievement,  attitude,  and  quantity  of 
interaction  when  adult  reentry  students  use  an  asynchronous  collaborative  learning  environment? 

3.  Do  positive  interdependence  and  affiliation  motive  interact  to  affect  achievement,  attitude,  and 
interaction  behavior  for  adult  reentry  students  in  an  asynchronous  collaborative  learning  environment? 

Method 

Participants  were  280  undergraduate  business  majors  enrolled  in  a required  course  in  management  at  a 
private  degree  completion  university  for  adult  learners.  Prior  to  the  study,  the  affiliation  scale  of  the  Work 
Motivation  Inventory  (Braskamp  & Maehr,  1987)  was  administered  to  measure  affiliation  motive.  Participants 
were  blocked  by  high  or  low  affiliation  motive,  then  randomly  assigned  to  one  of  four  small  group  treatments  — 
role  interdependence,  reward  interdependence,  role  plus  reward  interdependence,  or  no  interdependence. 

The  dependent  variables  were  achievement  and  attitude.  Achievement  was  measured  by  individual 
posttest  scores  following  asynchronous  online  instruction  and  practice.  Attitude  was  measured  by  a survey  that 
assessed  student  satisfaction  and  continuing  motivation  for  working  in  small  groups.  Interaction  behaviors  were 
observed  through  the  text  transcripts  of  group  work  and  qualitatively  categorized  according  to  an  observation 
protocol. 

Materials 

Three  instructional  units  were  derived  from  the  required  text  for  this  business  management  class.  Each 
unit  included  a learning  objective,  a 500-  750  word  online  lecture  with  an  open-ended  discussion  starter,  and  a 
practice  set.  The  practice  set  for  each  unit  provided  ten  selected-response  items,  including  eight  items  related  to 
concepts  and  two  application  items  related  to  a business  or  adult  education  scenario. 

The  instructor  notes  included  a weekly  schedule  for  delivering  the  sequentially  numbered  course 
components.  The  course  components  were  the  individual  text  -and-graphic  files  for  the  lectures,  practice  sets, 
and  directive  communications  to  the  participants. 

An  interaction  checklist  was  adapted  from  instruments  used  to  record  group  interactions  (Klein  & 
Schnackenberg,  2000;  Cavalier,  Klein  & Cavalier,  1995;  Hall  et  ah,  1988).  Adaptations  were  made  to  reflect 
verbal  behaviors  that  are  likely  to  occur  and  can  be  demonstrated  in  an  asynchronous  collaborative  learning 
environment  through  text  and  graphics  (Hillman,  1999;  Hall,  1997;  Kruger  et  ah,  1996).  A sample  of  interaction 
behaviors  from  40  triad  learning  teams  was  observed  through  the  text  transcripts  of  newsgroup  posts  made  by 
participants  during  group  work.  A single  post  made  by  one  student  could  contain  more  than  one  interaction 
behavior.  The  interaction  behaviors  were  qualitatively  analyzed  according  to  the  three  categories  of  cognitive, 
group  process,  and  off  task.  Cognitive  interactions  included  statements  about  course  topics,  such  as  discussing 
content  by  providing  examples  or  elaborating,  asking  questions,  answering  questions,  and  disputing  others’ 
opinions.  Group  process  interactions  included  statements  intended  to  accomplish  a task  by  interpreting 
requirements  or  establishing  due  dates;  managing  group  behavior  by  delegating,  accepting,  or  declining 
responsibility;  encouraging  team  mates;  and  commenting  on  the  experimental  project  in  which  they  were 
participating.  Off  task  interactions  included  statements  about  topics  not  related  to  this  course,  such  as  discussing 
self  or  others  in  a context  other  than  this  class  and  describing  events  not  related  to  the  course.  Inter-rater 
agreement  was  established  at  .91. 

Procedures 

All  procedures  took  place  in  an  asynchronous  environment  supported  by  Microsoft  Outlook 
Express  , which  is  the  usual  delivery  modality  for  participants.  Features  of  the  user  interface  and  the  activities 
that  take  place  are  referenced  with  names  that  mimic  typical  classrooms.  For  example,  when  participants  “go  to 
class,”  they  connect  to  the  Internet  and  gain  password  protected  access  to  files  for  their  registered  course.  When 
participants  “speak  up  in  class,”  they  submit  a text  message  that  is  displayed  chronologically  with  comments 
from  other  class  members,  available  for  all  to  read.  The  software  interface  organizes  text  -and-graphics  messages 
into  newsgroups  named  Main  Classroom,  Learning  Team  A,  Learning  Team  B,  etc.  Furthermore,  the  software 
interface  indents  replies  and  subsequent  comments,  creating  a visual  representation  of  the  interactions  taking 
place.  A topic  of  discussion  shown  with  its  associated  dialogue  is  called  a thread. 

Participants  communicated  with  one  another  by  posting  a message  to  either  the  Main  Classroom 
newsgroup  visible  to  all  members  of  the  class,  or  by  posting  a message  to  their  Learning  Team  visible  only  to 
the  small  group  members.  This  action  is  similar  to  sending  an  email  message,  except  that  the  recipient  is  a 
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group  of  people  rather  than  an  individual.  Participants  were  well  oriented  to  using  the  software  interface 
because  it  is  the  primary  means  of  communication  for  all  of  their  courses.  Additionally,  participants  had  been 
oriented  to  working  in  groups  during  a required  course  at  entry  to  their  academic  programs. 

The  instructor  posted  a script  in  each  triad’s  newsgroup  that  provided  guidelines  for  working  in  their  small 
group.  All  scripts  stated,  “Throughout  this  week,  use  the  lectures  and  practice  sets  to  prepare  for  the  test.” 
Additionally,  in  the  role  interdependence  condition,  each  group  of  three  participants  was  directed  to  designate 
the  roles  of  facilitator,  answer  drafter,  and  verifier  for  the  group  members.  Groups  were  prompted  to  rotate  roles 
for  each  unit.  In  the  rewaid  interdependence  condition,  each  group  of  three  participants  was  informed  that  they 
would  receive  a 5 -point  (15%)  bonus  toward  this  week’s  assignment  if  all  members  of  the  group  attained  a 
score  of  24  (80%)  or  more.  In  the  role -plus-reward  interdependence  condition,  each  group  of  three  participants 
was  directed  to  designate  roles,  and  was  informed  of  the  bonus  points  for  all  members  of  the  group  attaining  a 
score  of  80%  or  more.  In  the  no  structured  interdependence  condition,  each  group  of  three  participants  was 
informed  only  that  they  should  discuss  the  readings  and  questions  and  use  the  practice  sets  to  prepare  for  the 
test. 

At  the  end  of  the  week,  the  posttest  was  available  for  one  hour  in  order  to  simulate  a time-limited 
classroom  setting,  rather  than  an  open  book  test  that  could  be  taken  at  leisure.  Participants  were  told  to  take  the 
test  without  using  notes  or  reading  materials.  At  the  completion  of  the  posttest,  the  attitude  survey  was 
presented. 


Results  and  Discussion 

Achievement 

Results  indicated  no  significant  differences  in  achievement  by  type  of  interdependence  in  small  group 
work,  or  by  affiliation  motive  (see  Table  1).  Overall  achievement  was  about  71%  on  the  posttest,  a score  not 
unexpected  for  this  population.  A correlation  analysis  was  conducted  to  determine  the  relationship  between 
posttest  score  and  quantity  of  interactions  and  revealed  a significant  positive  correlation,  indicating  that 
participants  with  higher  numbers  of  interactions  attained  higher  posttest  scores. 

Participants  in  each  of  the  small  group  conditions  performed  similarly  on  the  posttest,  and  there  are  a 
couple  of  possible  reasons  for  this  result.  Although  positive  interdependence  was  structured  in  various  ways  in 
this  study,  other  elements  of  cooperative  learning  were  present  in  all  treatments.  Participants  in  all  small  groups 
had  individual  accountability  and  computer- mediated  promotive  interaction.  Furthermore,  while  group 
facilitation  skills  and  evaluation  of  interpersonal  and  group  processes  were  not  specifically  directed  in  the 
current  study,  participants  had  previously  been  trained  on  these  and  other  elements  of  cooperative  learning.  In 
fact,  all  students  who  enroll  at  the  university  where  the  study  was  implemented  are  trained  on  how  to 
collaborate  at  the  start  of  their  degree  program. 

The  current  study  points  to  the  challenge  of  isolating  positive  interdependence  from  other  elements 
known  to  be  integral  to  cooperative  learning  (Johnson  & Johnson,  1998b).  Previous  research  on  cooperative  and 
small  group  learning  has  focused  on  comparing  individual  versus  group  work,  and  only  recently  has  the  relative 
effectiveness  of  various  structures  of  positive  interdependence  been  documented  (Jensen,  Johnson,  & Johnson, 
2002).  Results  suggest  that  structuring  positive  interdependence  by  itself  does  not  affect  achievement  when 
college  reentry  students  work  in  an  asynchronous  cooperative  learning  environment. 

Secondly,  the  instructional  materials  used  by  participants  in  all  treatment  groups  were  designed  with 
stated  objectives  and  alignment  of  the  lesson  content,  practice  exercises,  and  posttest.  Consideration  must  be 
given  to  Bossert’s  (1988-89)  assertion  that  researchers  comparing  individual  and  cooperative  learning  do  not 
consistently  find  differences  between  these  methods  when  well-designed  instructional  materials  are  used.  The 
effect  of  structuring  positive  interdependence  is  likely  to  be  weak  when  well-designed  instructional  materials 
are  used. 

Posttest  achievement  may  have  been  improved  with  the  use  of  specific  feedback  from  the  instructor  to 
students.  Kruger  et  al.  (1996)  found  that,  in  both  face-to-face  and  computer- mediated  adult  training 
environments,  messages  from  the  expert  were  rated  as  most  valuable  in  learning  to  accomplish  the  task. 
Furthermore,  specific  feedback  was  more  highly  rated  than  global  feedback.  In  the  current  study,  practice 
exercises  were  debriefed  by  group  members,  but  without  feedback  or  remediation  from  the  instructor. 
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Table  1 Means  and  Standard  Deviations  for  Achievement  Posttest  Scores  by  Small  Group  Treatments  and 
Affiliation  Motive. 


Type  of  Interdependence 


Role 

Reward 

Both 

None 

Total 

Affiliation  Motive 
High 

22.40 
3.49 
(n  = 35) 

21.95 
3.80 
(n  = 38) 

20.97 
3.87 
(n  = 41) 

20.47 
3.66 
(n  = 30) 

21.47 
3.76 
(n  = 144) 
72% 

Low 

21.24 
3.48 
(n  = 38) 

20.34 
3.77 
(n  = 35) 

21.52 
3.49 
(n  = 42) 

20.77 
3.37 
(n  = 30) 

21.01 
3.53 
(n  = 145) 
70% 

All 

21.79 
3.51 
(n  = 73) 
73% 

21.18 
3.85 
(n  = 73) 
71% 

21.25 
3.67 
(n  = 83) 
71% 

20.62 
3.49 
(n  = 60) 
69% 

21.24 
3.64 
(n  = 289) 
71% 

*Note:  Total  possible  score  was  30. 


Attitude 

Participants  in  the  reward  condition  had  significantly  higher  agreement  with  several  attitude  statements 
that  reflected  benefit  from  working  with  others,  being  able  to  generate  better  ideas  than  they  could  have  done  as 
individuals,  and  the  importance  of  fellow  team  mates  earning  a high  score  (see  Table  2).  Furthermore, 
participants  in  all  three  types  of  structured  interdependence  groups  (roles,  reward,  and  roles -plus-reward)  had 
significantly  higher  agreement  with  the  statement  "I  was  able  to  learn  more  because  my  team  members  knew  it 
was  their  job  to  contribute  to  the  group  work,"  compared  to  groups  with  no  structured  interdependence. 

Kagan  (1994)  describes  positive  interdependence  as  the  “subjective  experience  of  being  on  the  same 
side”  (p.  129).  The  finding  that  participants  in  the  no  structured  interdependence  condition  had  lower  attitude 
ratings  corroborates  previous  research  identifying  reward  structures  that  increase  the  perception  of  positive 
interdependence  and  reduce  social  loafing  (Johnson,  Johnson,  & Holubec,  1994;  Webb,  1997).  In  particular,  the 
identified  reward  structures  are  ones  that  combine  the  points  attained  by  each  member’s  contributions,  or  offer  a 
bonus  when  all  members  exceed  a standard.  Similarly,  the  reward  structure  in  the  current  study  provided  that 
team  members  would  receive  a bonus  if  each  member  attained  a criterion  score. 

Furthermore,  concern  for  teammates’  success  was  apparently  facilitated  by  reward  structures.  Attitude 
statements  regarding  the  importance  of  team  members  earning  a high  score  and  every  member  being  highly 
successful  were  significantly  higher  in  the  conditions  of  reward  and  role  -plus-reward,  compared  to  the  no 
structured  interdependence  condition.  There  is  considerable  support  for  this  finding,  including  Slavin’s  (1991) 
assertion  that  group  rewards  are  essential  to  eliciting  concern  for  group  mates’  achievement.  Previous  empirical 
evidence  indicates  students’  greater  willingness  to  explain  the  material  and  ask  questions  (Cole  & Smith,  1993) 
as  well  as  higher  peer  evaluations  for  group  members  when  rewards  were  used  cooperatively  rather  than 
competitively  (Niehoff  & Mesch,  1991). 

The  effect  of  role  interdependence  in  this  study  is  demonstrated  in  the  responses  to  the  attitude 
statement,  “I  was  able  to  learn  more  because  my  team  members  knew  it  was  their  job  to  contribute  to  the  group 
work,”  with  role  groups  having  significantly  higher  agreement  than  groups  with  no  stmctured  interdependence. 
Table  2 Means  for  Attitude  Item  Responses  by  Small  Group  Treatments. 


143 


Type  of  Interdependence 


Role 

Reward 

Both 

None 

Total 

1. 1 would  have  enjoyed  this  activity  more  if  I 
had  worked  by  myself. 

1.78 

1.60 

1.99 

1.70 

1.78 

2.  Working  with  other  students  in  this  type  of 
activity  encourages  me  to  stick  with  my  degree 
program. 

2.96 

3.35 

3.00 

2.92 

3.06 

3.  I benefited  from  working  with  others  during 
these  lessons.  * 

3.46 

3.50 

3.42 

2.80 

3.32 

4.  As  a group,  we  generated  better  ideas  than 
we  could  have  done  as  individuals.  * 

3.36 

3.69 

3.40 

2.93 

3.35 

5.  At  the  start  of  the  week,  I knew  how  to 
interact  with  my  team  members. 

3.05 

3.05 

3.09 

3.10 

3.07 

6.  I was  able  to  learn  more  because  I knew  it 
was  my  job  to  help  the  other  team  members 
understand  the  material. 

3.02 

2.83 

3.05 

2.62 

2.90 

7. 1 was  able  to  learn  more  because  my  team 
members  knew  it  was  their  job  to  contribute  to 
the  group  work.  * 

3.00 

3.06 

2.98 

2.32 

2.87 

8.  In  future  team  activities,  I would  prefer  that 
each  member  of  my  team  be  assigned  specific 
roles. 

3.01 

2.59 

2.63 

2.77 

2.75 

9.  During  this  week,  it  was  important  to  me  that 
every  team  member  earned  a high  score.  * 

3.99 

4.56 

4.39 

3.75 

4.19 

10.  For  these  lessons,  it  was  important  to  me 
that  every  member  was  highly  successful.  * 

4.06 

4.59 

4.33 

3.75 

4.20 

1 1 . Thinking  about  my  score  on  the  final  exam 
helped  me  work  with  the  team  members. 

3.56 

3.62 

3.56 

3.07 

3.47 

12.  In  future  team  activities,  I would  prefer  to 
work  for  points  based  on  my  team  members’ 
performance. 

2.26 

2.10 

2.39 

2.00 

2.20 

Note:  Likert  type  scale  for  responses  consisted  of  5 = Very  true,  4 = Mostly  true,  3 = Moderately  true,  2 
Slightly  true,  1 = Not  true. 

*significant  at  p < .01 
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Previous  research  points  to  increased  learner  satisfaction  when  using  defined  roles  in  small  group  work 
(Cavalier,  Klein,  & Cavalier,  1995;  O'Donnell  et  al,1987). 

When  differences  in  attitude  are  examined  by  affiliation  motive,  participants  with  high  affiliation 
motive  had  significantly  higher  agreement  with  six  of  the  twelve  attitude  statements.  This  result  may  be 
reflective  of  a general  preference  for  group  work,  as  noted  in  previous  research  on  affiliation  motive  in  small 
group  work  (Chan,  1980;  Hall  et  ah,  1988;  Klein  & Schnackenberg,  2000;  Brewer,  Klein,  & Mann,  2003). 
Likewise,  high  affiliation  participants’  agreement  with  attitude  statements  may  be  reflective  of  a personality 
trait,  or  predisposition  to  act  cooperatively  (McClelland,  1976;  Johnson,  Johnson,  & Smith,  1998a;  Huitt,  2001). 

Interactions 

Overall,  individual  participants  in  groups  with  role -plus-reward  interdependence  had  the  highest 
number  of  interactions,  statistically  significant  compared  to  those  in  groups  with  only  reward  interdependence 
or  no  structured  interdependence.  This  result  supports  previous  findings  from  studies  using  roles  and  from  other 
studies  that  investigated  the  use  of  rewards.  Adults  using  roles  during  group  work  had  increased  interactions 
(Cavalier,  Klein,  & Cavalier,  1995;  Klein  & Doran,  1999),  while  college  students  with  implicit  incentives  for 
cooperative  group  work  used  more  elaboration  and  metacognitive  strategies  (Karabenick  & Collins-Eaglin, 
1997).  The  importance  of  increased  interactions  is  noted  in  previous  findings  that  increased  collaboration  results 
in  better  problem  solving  (Flynn  & Klein,  2001;  Uribe  & Klein,  2003). 

Looking  at  the  specific  nature  of  the  interactions,  there  were  significant  differences  in  the  frequencies 
of  cognitive,  group  process,  and  off  task  interactions  among  the  four  treatment  conditions  (see  Table  3).  Groups 
with  no  structured  interdependence  had  the  highest  number  of  cognitive  interactions,  groups  with  role 
interdependence  had  the  highest  number  of  group  process  interactions,  and  groups  with  reward  interdependence 
had  highest  number  of  off  task  interactions. 

In  light  of  these  combined  findings,  it  appears  that  there  is  an  advantage  to  using  multiple  avenues  of 
interdependence  for  increasing  interactions.  According  to  Webb  (1997),  learning  theorists  generally  agree  that 
students  learn  most  by  participating  actively  in  group  work.  Conversely,  passive  behavior  may  have  minimal 
consequences  for  a group  outcome,  but  is  detrimental  to  individual  learning  (Webb,  1993).  Passive  behavior  is 
observed  in  predictable  patterns  known  as  social  loafing  and  free  rider  and  sucker  effects  (Slavin,  1990;  Karan 
& Williams,  1993). 


Implications 

The  sample  in  the  current  study  was  representative  of  the  target  population  of  adult  reentry  students, 
and  implications  are  directed  at  that  group  of  students.  The  similar  distribution  of  posttest  scores  and  low 
variability  across  all  treatment  conditions  point  to  a normally  distributed  population.  The  median  score  on  the 
need  for  affiliation  measure  was  about  the  same  as  the  average  noted  for  the  general  adult  population. 

The  attitude  findings  in  this  study  support  the  recommendation  to  use  small  groups  with  adult  reentry 
students.  Overall,  participants  did  not  want  to  work  alone,  as  evidenced  by  the  low  agreement  across  all  groups 
with  the  attitude  item,  “I  would  have  enjoyed  this  activity  more  if  I had  worked  by  myself.”  This  preference  for 
group  work,  even  when  there  is  no  increased  achievement,  has  been  previously  demonstrated  in  other  studies 
(Palinscar  & Brown,  1989;  Klein  & Pridemore,  1992;  Thompson  & Scheckley,  1997;  Klein  & Doran,  1999; 
Brewer,  Klein,  & Mann,  2003). 

The  positive  relationship  of  interactions  and  achievement  was  established  in  this  study.  On  the  one 
hand,  high  interactivity  may  be  a feature  of  top  performing  students.  On  the  other  hand,  the  importance  of 
interactions  may  be  even  greater  in  an  asynchronous  computer-mediated  environment  corrpared  to  a face-to- 
face  classroom  (Gunawardena,  1995;  Bailey  & Luetkehans,  1998).  In  the  absence  of  visual,  auditory,  and  tactile 
communication  cues,  text  based  interactions  constitute  the  entirety  of  online  class  communication  (Hsu  & 
Sammons,  1998).  Given  that  interactivity  is  a vital  component  of  asynchronous  classrooms,  educators  and 
instructional  designers  should  provide  structures  that  maximize  meaningful  exchanges  among  participants. 

Beyond  merely  increasing  the  frequency  of  interactions,  consideration  should  be  given  to  the  nature  of 
these  interactions.  In  the  current  study,  participants  with  structured  interdependence  had  better  attitudes  about 
learning  with  the  contributions  of  their  team  members.  The  cooperative  behavior  of  taking  turns  does  not  exist 
in  an  asynchronous  environment  where  the  messaging  software  delivers  participants’  contributions  in  the  time- 
stamped  order  in  which  they  arrived. 
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Table  3 Summary  of  Results  on  Nature  of  Interactions  by  Small  Group  Treatments. 


Treatment 

Result 

Interaction 

Sample  comment 

Role 

Role  interdependence  groups  had 
more  group  process  interactions 
than  did  reward  groups.  * 

Group 

Process 

I'm  not  sure  if  we  are  going  to  be  able  to 
count  on  a third  team  member.  Let's 
proceed  as  follows.  Whether  we  get  a 
response  from  R.  or  not  we  need  to 
complete  all  questions,  of  which  8 have 
already  been  verified.  Finally,  the 
summary  needs  to  be  completed,  which 
I'll  do  tonight. 

Reward 

Reward  interdependence  groups  had 
more  off  task  behaviors  than  did 
role  and  role  -plus-reward 
interdependence  groups.  * 

Off  Task 

Thanks,  J.,  a lot  of  rain  here  also,  but  my 
flowers  needed  it.  I’m  hoping  the 
humidity  drops. 

Both 

Role -plus-reward  (both) 
interdependence  groups  had  more 
group  process  interactions  than  did 
the  reward  group.  * 

Group 

Process 

Here's  my  thoughts  towards  session  2. 
Open  for  discussion  of  course.  Please  post 
the  unitl  final  answers,  I didn't  quite 
finish  moving  my  oldest  two  daughters  to 
Baltimore.  I won't  be  back  online  until 
after6PMEDT.  Thanks  in  advance. 

None 

No  structured  interdependence 
groups  had  more  cognitive 
interactions  than  reward  only  or  role 
only  groups.  * 

Cognitive 

To  an  outsider,  it  would  seem  that  you  work 
for  a division  of  XYZ.  Not  being  an  expert 
on  XYZ,  I know  they  provide  many 
services,  e.g.  disaster  relief,  fund  raising.  It 
would  be  conceivable  to  consider  your 
organization  a functional  structure.  What  do 
think? 

Note:  Interactions  by  10  triad-groups  in  each  treatment. 

* significant  at  p < .01 

Even  when  messages  are  displayed  together  for  a single  topic,  the  synchronous  quality  of  having  an 
utterance  associated  with  the  one  immediately  preceding  it,  and  the  feedback  loop  provided  therein,  does  not 
exist  in  computer- mediated  communications  (Hillman,  1999).  Therefore,  assigning  roles  for  beginning  and 
ending  conversations  may  be  beneficial. 

Looking  further  at  the  nature  of  cooperative  interactions,  it  is  important  that  adult  learners  feel  that 
their  efforts  during  group  work  were  worthwhile.  In  the  current  study,  participants  in  reward  groups  had 
significantly  higher  attitude  ratings  for  the  item,  “As  a group,  we  generated  better  ideas  than  we  could  have 
done  as  individuals,”  compared  to  those  with  no  structured  interdependence.  Teachers  should  provide  tasks  and 
sufficient  incentives  to  encourage  the  type  of  collaboration  that  results  in  valuable  contributions. 

The  finding  that  participants  with  no  structured  interdependence  had  significantly  more  cognitive 
interactions  than  either  role  only  or  reward  only  is  noteworthy  for  educators.  When  teachers  of  adult  learners 
use  group  work,  they  should  ensure  that  there  is  opportunity  for  students  to  ask  and  answer  questions  within  the 
groups.  Moreover,  it  may  be  less  necessary  for  an  instructor  to  structure  positive  interdependence  when  other 
elements  of  cooperative  learning  are  present. 
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Further  Research 

Further  research  should  investigate  the  quantity  and  nature  of  interactions  in  asynchronous,  computer- 
mediated  environments.  Specifically,  studies  should  be  conducted  to  determine  whether  increasing  interactions, 
or  increasing  particular  types  of  interactions,  is  beneficial  to  learning  and  persistence.  Future  research  on 
affiliation  motive  in  small  group  work  may  clarify  whether  the  higher  attitude  ratings  found  in  this  study  were 
reflective  of  a general  preference  for  group  work  by  high  affiliation  adults,  or  if  group  settings  do  indeed 
provide  an  optimal  environment  for  some  learners,  but  not  for  others.  The  current  study  did  not  compare  group 
versus  individual  work  in  the  asynchronous  online  environment,  and  this  aspect  should  be  examined  in  future 
studies,  focusing  on  both  achievement  and  attitude. 

Additionally,  the  use  of  teacher  intervention  and  feedback  during  group  work  should  be  explored. 
Although  the  current  study  systematically  excluded  teacher  feedback,  caution  should  be  used  in  guarding 
against  the  implication  that  copious  student  interaction  is  sufficient.  Rather,  consideration  should  be  given  to  the 
conclusions  of  Kruger  et  al.  (1996)  that  the  free  sharing  of  ideas  that  can  take  place  in  computer-mediated 
communication  does  not  exclude  the  need  for  expert  feedback. 

Findings  from  the  current  study  do  not  support  the  assertion  that  structured  positive  interdependence 
can  affect  achievement  in  a computer- mediated  environment  (Jensen,  Johnson,  & Johnson,  2002),  but  the  body 
of  literature  regarding  cooperative  learning  in  this  medium  is  sparse.  In  light  of  the  growing  prevalence  of 
online  distance  programs  for  adult  learners,  it  is  worthwhile  to  continue  exploring  whether  previous 
assumptions  about  cooperative  learning  in  face-to-face  environments  are  replicated  in  asynchronous,  computer- 
mediated,  and  distance  learning  settings.  Research  of  this  type  may  help  us  better  understand  the  conditions 
under  which  cooperative  learning  is  most  effective  for  adults  in  online  environments. 
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Design,  Development,  and  Evaluation  of  Electronic  Portfolios  for 
Advanced  Degree  Programs  in  Technology  and  School  Media 
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Abstract 

Across  the  various  programs  of  study,  portfolios  serve  many  functions  and  purposes.  Electronic 
portfolios  have  been  used  as  a culminating  product  for  students  in  our  Master  of  Library  Science  and  Master  of 
Education  in  Instructional  Technology  at  East  Carolina  University.  Two  conclusions  were  drawn  from  this 
study  - (1 ) authentic  assessment  using  portfolios  is  useful  for  facilitating  reflective  thinking  that  results  in  self- 
regulated  learning,  and  (2)  student  products  in  the  form  of  electronic  files  can  be  archived,  indexed  and  used  as 
evidences  in  program  evaluation. 


Introduction 

The  need  for  authentic  methods  for  assessing  educational  outcomes  has  led  to  a move  from 
quantitative  measures  to  a more  an  open-ended  qualitative  format.  Through  the  use  of  portfolios,  students  are 
able  to  select  and  evaluate  their  own  products  of  learning  and  present  these  for  final  certification  before  entering 
their  respective  professions.  The  portfolio  provides,  not  only  a method  for  assessment,  but  also  is  a catalyst  for 
learning.  These  measures  are  dependent  on  process,  as  well  as  outcomes  from  the  learning  ejperience.  Using 
the  processes  for  selection,  evaluation,  and  alignment  of  work  samples  with  a particular  standard  for  excellence 
would  likely  help  the  student  transfer  what  is  learned  within  the  laboratory  setting  to  the  real  world  of  work 
(Wolf,  1998). 


Portfolio  Formats  and  Functions 

One  of  the  earliest  reported  uses  for  the  portfolio  was  in  the  visual  arts  for  the  purposeful  collection  of 
one’s  best  work  (Friedman  Ben  David,  et  ah,  2001).  Many  schools  continue  the  use  of  the  showcase  portfolio 
to  display  exemplary  student  products  as  a culminating  experience  and  for  prospective  employers  (Baltimore  & 
Hickson,  1996).  When  using  this  type  of  showcase  portfolio,  there  are  no  comparisons  between  entry-level 
work  and  expert  performance  by  the  student.  Thus,  the  contents  do  not  provide  substantial  evidence  that  training 
or  education  has  had  notable  impact  on  the  student. 

Because  of  our  society’s  evolution  from  industrial-  lockstep  work  environments  to  more  open-ended- 
informational  environments,  there  is  an  important  emphasis  on  authenticity  in  student  learning  experiences. 
Employers  are  looking  for  candidates  who  can  examine  their  environment,  draw  logical  conclusions,  and 
develop  problem-solving  strategies  based  on  a given  situation  (Weiner,  2000).  This  is  accomplished  through  the 
use  of  situated  problem  solving  and  authentic  assessment  of  outcomes  from  learning  (Young,  1995). 

Another  important  use  of  portfolio  is  program  evaluation.  Portfolio  assessment  requires  the  careful 
analysis  of  program  goals  and  objectives  and  how  these  are  transferred  to  the  course  activities  and  assignments. 
Student  artifacts  should  mirror  program  goals.  When  this  is  evident  within  the  portfolio,  evaluation  of  the 
program  of  study  is  facilitated  (Koretz,  1992;  Payne,  2000). 

Portfolios  and  Learning 

Regardless  of  format,  function,  or  purpose,  portfolios  can  be  classified  as  either  capstone  experience 
(showcase)  or  a record  of  process  in  learning  (assessment  portfolio).  The  capstone  portfolio  includes  stand- 
alone evidence  for  mastery  of  program  objectives,  examples  of  student’s  best  work,  and  documents  from 
culminating  experiences.  Typically,  accomplished  students  who  are  about  to  enter  their  chosen  profession  are 
associated  with  the  "capstone"  category.  Programs  that  require  the  capstone  (or  showcase)  portfolio  should 
specify  work  samples  that  will  be  of  interest  to  prospective  employers  and  artifacts  that  are  cognizant  of  the 
profession.  In  addition,  expectations  and  standards  for  best  practices  must  be  clearly  communicated  to  the 
student  (Skawinski  & Thibodeau  2002). 

A second  category  is  the  "process  or  learning  portfolio.”  The  contents  represent  processes  for  cognitive  growth, 
interrogation  of  the  learning  environment,  self-assessment  using  recognized  standards,  and  transference  of 
learning  to  the  workplace.  For  the  instructor  or  faculty  member,  there  is  a responsibility  to  the  student  to 
monitor  cognitive  growth  as  a result  of  assigned  projects  and  field  experiences.  By  providing  cognitive 
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scaffolds  for  reflection,  self-assessment,  and  strategies  for  making  changes,  the  process  approach  to  learning  is 
exemplified  within  the  contents  of  the  portfolio  (Murphy,  1997).  The  instructor/assessor  acts  as  a guide  or 
proctor  during  the  development  of  the  portfolio,  and  models  collaborative  practices  as  mentor  and  mentee  work 
together  to  select  artifacts  and  other  evidence  that  show  growth  over  time.  An  important  role  of  the 
instructor/assessor  is  to  provide  critical  commentary  and  invite  the  student  to  defend,  justify,  and  make 
adaptations  to  his  or  her  work  samples.  The  dialog  between  faculty  and  student  can  be  very  productive  during 
these  advising  sessions. 

In  addition  to  evaluation  of  student  learning,  either  process  or  capstone  portfolios,  are  useful  for  the 
analysis  of  a program  of  studies.  Program  evaluation  is  facilitated  when  key  players  for  the  portfolio  process  are 
committed  to  the  necessary  time  requirements,  practices  for  self-evaluation,  and  adoption  of  authentic 
assessment  methodologies  (Johnson,  et  ah,  2000;  Campbell,  et  al.,  2000;  Baume  & Yorke,  2002).  For  the 
student,  this  means  gaining  skills  as  a reflective  practitioner.  He  or  she  must  be  willing  to  adopt  the  process- 
approach  for  learning.  This  means  entering  the  program  as  a novice,  accepting  critical  commentary,  working 
through  revisions,  and  planning  for  the  future.  For  the  assessor,  it  requires  a commitment  for  adequate  time  with 
students  for  mentoring  and  modeling  for  reflective  practices  (Freidus,  1996).  In  addition,  there  must  be  time 
devoted  to  careful  planning  for  program  goals,  objectives,  and  classroom  activities  that  reflect  these  objectives. 

Reliability  and  Validity  Using  Assessment  Portfolios 

A major  consideration  with  implementation  of  portfolio  assessment  is  reliability  of  measures  and 
validity  of  the  assessment.  Latrobe  and  Lester  (2000)  discovered  in  their  Library  Science  program  that 
establishing  valid  measures  is  difficult  because  competent  performance  may...  “vary  in  depth,  in  approach,  and 

in  the  specificity  of  the  professional  work  addressed ”,  Although  it  is  difficult  to  gather  data  related  to 

reliability  in  portfolio  assessment,  (Friedman  Ben-Davis  & et  al.,  2001)  as  a result  of  this  review,  several  studies 
were  identified,  and  were  supportive  of,  portfolios  for  assessment  purposes  (Baume  & Yorke,  2002).  Other 
reports  are  not  as  encouraging  (Koretz,  1998).  There  are,  however,  certain  characteristics  that  were  apparent  in 
programs  with  reports  for  reliable  use  of  portfolio  assessment.  Measures  are  reliable  when  there  is  evidence  that 
portfolio  contents  represent  an  accurate  picture  of  the  program  goals/objectives  or  other  recognized  standards 
for  the  profession  (Bullock  & Hawk,  2001;  Campbell,  et  al.,  2000;  Pitts,  Coles,  & Thomas,  2001;  Routledge  & 
Willson,  1997).  In  addition,  correlations  among  assessors’  scores  are  high  when  there  is  evidence  for  clear-cut 
indicators  of  acceptable  performance.  Another  characteristic  associated  with  the  reliable  use  of  portfolios  is  the 
selection  of  artifacts;  either  specified  in  advanced  or  self-selected  by  the  student,  these  should  be  representative 
of  program  goals  and  objectives.  Along  with  specific  criteria,  there  are  standardized  levels  of  difficulty  and 
consistency  in  characteristics  of  the  evidence  or  artifacts.  Reports  from  the  literature  suggest  that  correlations 
can  be  very  low  when  there  are  inconsistencies  among  artifacts.  Reliability  measures  were  high  when  clear-cut 
criteria  for  evaluation  had  been  agreed  upon  by  assessors  and  performance  indicators  were  representative  of  the 
standards  or  competencies  adopted  by  the  program.  Reliability  measures  were  also  high  with  reports  for 
sufficient  training  of  assessors. 

One  strategy  used  by  programs  to  ensure  strong  reliability  and  validity  measures  was  through 
collaborative  meetings  to  reach  consensus  on  scores.  When  planning  implementation  of  portfolios,  faculty 
should  meet  to  analyze  the  strength  of  relationships  between  program  goals,  performance  indicators,  and  quality 
of  the  portfolio  contents.  Typically,  there  are  three  assessors  assigned  to  a team.  Contents  are  evaluated  by  the 
first  2 assessors  who  score  independently.  When  there  are  wide  differences  in  scoring,  a third  assessor  reads  and 
evaluates  only  those  sections  with  disparate  scores.  ( Baume  & Yorke,  2002;  Friedman  Ben  David,  et  al,  2001; 
Davis,  et  al.,  2001;  Skawinski  & Thibodeau,  2002)  Of  the  studies  reviewed,  the  third  reader  usually  scores  in 
agreement  with  “pass”  or  “marginally  pass”.  Careful  alignment  of  program  objectives  with  course  activities, 
clear  communication  of  expectations  aligned  with  these  objectives,  and  a specified  standard  for  formatting  and 
presentation  of  the  portfolio  were  all  associated  with  valid  and  reliable  measures. 

Purpose  of  the  Study 

The  purpose  of  this  study  was  to  identify  specific  advantages  for  the  use  of  assessment  portfolios  and 
how  these  advantages  might  support  the  graduate  programs'  goals  and  objectives.  To  satisfy  these  purposes,  a 
formative  evaluation  of  student  reflections  and  faculty  satisfaction  for  use  of  portfolio  as  a method  for 
assessment  was  conducted.  Three  main  areas  were  the  focus  of  the  study-  student  response  to  portfolios  with 
reflective  writing  as  a metacognitive  process,  validity  of  the  evaluation  methods,  and  solutions  to  management 
of  data  for  program  evaluation. 
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Self-regulated  Learning  and  Reflective  Writing 

As  a method  for  evaluating  student  response  to  electronic  portfolios,  end  of  course  surveys  were  sent 
to  students  enrolled  in  the  graduate  level  course  "Development  of  Electronic  Portfolios”.  The  course  was 
offered  as  an  elective  for  students  enrolled  in  any  of  the  department's  three  graduate  programs.  The  two  main 
objectives  for  the  course  were  (1)  guide  students  through  the  reflective  writing  process,  and  (2)  teach  skills  for 
file  management  and  design  of  the  portfolio  using  web  editors.  Students  also  increased  their  comfort  level  in  the 
use  of  tools  for  electronic  file  transfer  and  online  development  of  the  course  projects.  The  course  received  high 
evaluations  and  has  been  requested  for  subsequent  semesters.  There  are  two  reasons  for  this  . First,  students 
recognized  the  need  for  developing  their  portfolios  with  skill,  accuracy,  and  as  a true  measure  of  academic 
achievement.  The  course  seemed  to  meet  this  need.  Secondly,  students  were  able  to  interact  with  the  instructor 
(also  their  faculty  advisors),  and  with  each  other,  for  feedback  on  their  progress  in  developing  the  portfolio. 
Dialog  was  constructive  within  a nonthreatening  environment  thus  students  experienced  a formative  assessment 
of  their  work  contributing  to  greater  confidence  for  completing  the  portfolio. 

An  important  concept  built  within  the  course  was  self-regulated  learning  using  the  reflective  writings 
for  each  artifact.  Self-regulation  as  a method  for  achieving  learning  goals  leads  to  increased  motivation,  self- 
monitoring, attention  control,  application  of  learning  strategies,  and  other  metacognitive  thinking  processes 
(Ormrod,  1999).  Using  a theoretical  base  for  self -regulated  learning  (Zimmerman  & Bandura,  1994),  questions 
for  the  survey  were  designed  to  gather  information  in  how  students  used  the  development  of  their  portfolios  and 
reflective  writings  for  the  following  thinking  processes  - focus  their  thinking  and  goal-setting,  self-assessment 
of  quality  of  work  based  on  standards,  and  time  management  strategies.  Table  1 provides  a summary  of 
comments  related  to  each  main  area  of  self-regulated  learning. 

Table  1.  Student  comments  related  to  self-regulated  learning  and  use  of  portfolios. 

For  focused  thinking,  students  commented: 

Writing  the  reflection  is  usually  a pretty  long  process  for  me  that  takes  a few  rough 
drafts  before  the  final  draft  is  complete.  It  is  almost  like  I am  constantly  reflecting 
on  my  reflection,  if  that  makes  any  sense.  Hopefully  I will  become  better  and  more 
focused  on  writing  my  reflections  with  more  experience  of  doing  so.  ...  The 
portfolio  definitely  forces  one  to  focus  upon  what  one  has  accomplished.  The 
reflective  writings,  in  particular,  has  helped  me  to  focus  on  what  I did,  why  it  was 
important,  how  it  could  be  improved,  and  how  my  future  will  be  impacted  by  what  I 
did. 

For  self-assessment  of  progress,  students  commented: 

I do  think  that  my  portfolio  will  help  me  assess  my  progress  as  I complete  more 
courses  during  my  degree.  Hopefully  the  quality  of  my  artifacts  and  reflections  will 
improve  each  course  and  by  adding  those  to  my  portfolio  will  help  me  see  if  this 
happens  or  not.  I also  think  that  my  portfolio  layout  will  probably  change  over  the 
course  of  completing  my  degree.  I will  reflect  as  well  on  the  layout  and  design  to 
hopefully  make  it  look  even  better  and  more  professional  for  others  get  a clearer 
picture  of  who  I am  and  what  I have  to  offer.  . . . The  reflective  writing  process 
enabled  me  to  assess  my  progress  (where  I’ve  been)  and  decide  the  future  path  to 
take  (where  I want  to  go)..  . . Through  reflection  I have  learned  to  assess  the 
strengths  and  weaknesses  of  a project,  which  in  turn  helps  me  decide  how  I can  plan 
the  next  project  more  efficiently. 

For  Time -management,  students  commented: 

I do  feel  that  the  portfolio  has  contributed  to  my  planning  and  organization,  but  do 
not  feel  the  reflections  have  done  the  same.  . . I would  have  completed  the  portfolio 
in  a timely  manner  regardless  of  disciplined  planning.  I have  difficulty  with  time 
management,  and  I cannot  hones  tly  say  that  the  portfolio  has  contributed  to  better 
organization.  . . . the  reflective  writing  did  not  affect  time  management. 

Students  reported  less  than  favorable  improvement  in  time  management  associated  with 
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reflective  writing  and  the  portfolio  process.  It  likely  that  additional  time  needed  to  learn  new  skills  for 
developing  the  electronic  portfolio  had  a negative  effect  on  their  perceptions  for  time  management.  In 
addition,  students  may  have  misinterpreted  the  survey  question  and  answered  as  time  required  to 
develop  the  portfolio  rather  than  the  actual  intent  - improved  time  management  skills  for  course 
assignments  as  a result  of  the  reflective  writings. 

Reflective  Writings  for  Self-assessment 

Each  artifact  in  a student's  portfolio  must  be  aligned  with  a standard,  professional  competency,  or 
program  objective.  Program  areas  vary  in  their  decision  to  allow  students'  self-selection  of  artifacts.  Reports  in 
the  literature  suggest  student  self-selection  is  beneficial  for  self-assessment  and  monitoring  of  one's  own 
learning.  However,  for  program  evaluation,  our  program  area  faculty  have  consistently  recommended  pre- 
selected artifacts  that  can  be  archived  and  used  for  program  evaluation  and  accreditation  purposes.  Each  core 
course  within  their  degree  program  includes  an  assignment  that  is  designated  for  the  student's  final  portfolio. 

The  student  must  analyze  the  requirements  and  final  outcomes  from  the  assignment  to  determine  appropriate 
standards,  competencies,  or  objectives.  Students  are  taught  to  describe  the  what,  how,  and  where  of  an 
assignment,  describe  outcomes  ("are  you  satisfied,  what  would  you  change  about  your  final  product?")  and  to 
align  their  assigned  work  sample  as  evidence  for  meeting  the  standard,  objective,  or  competency.  Students  must 
also  reflect  on  how  the  assignment  will  impact  their  future  career  goals.  The  rationale  for  including  this 
requirement  to  the  reflection  is  to  aid  in  transfer  of  skills  and  concepts  to  future  work  environments.  In  Table  2. 
is  a student's  reflective  writing  that  provides  clear  and  consistent  evidence  that  a competency  has  been  met  and 
that  transfer  to  the  workplace  is  highly  probable. 

Table  2.  Sample  of  Student  Reflective  Writing. 

LIBS  6014  Introduction  to  Reference.  Reflection  MLS  Program  Objective  3:To  answer 
reference  questions  using  print  and  electronic  resources 

In  LIBS  6014,  students  were  asked  to  “compile  a pathfinder  of  reference  sources  on  a specific 
topic  designed  for  a specific  user  group  and  based  on  a review  of  the  existing  literature  in  any 
given  subject  field.’'  I chose  to  create  a pathfinder  on  plants  since  I am  teach  third  grade 
Science,  and  this  would  be  useful  to  me  in  my  classroom  since  the  North  Carolina  Standard 
Course  of  Study  includes  plant  adaptation  and  growth  as  Goal  1 in  third  grade.  . . . Since 
creating  this  pathfinder,  I have  been  compiling  pathfinders  on  other  thematic  units  that  I 
currently  teach  as  well  as  others  that  I think  will  help  my  colleagues.  I have  shared  my 
pathfinders  with  my  colleagues  and  media  specialist  at  my  school.  Whenever  a teacher  begins 
a new  thematic  unit,  they  now  come  to  me  for  guidance. 

Artifact:  Pathfinder  for  6018: 

"Election  Year  Politics:  The  Political  Right  vs.  the  Political  Left:  What  Does  it  All  Mean?" 

This  project  exemplifies  MLS  Objective  3:  To  answer  reference  questions  using  print  and  electronic 
resources 

A pathfinder  is  a useful  tool  that  points  people  to  sources  of  information.  I chose  this  topic 
because  it  is  very  relevant  to  American  culture  in  this  election  year.  This  pathfinder  will  help 
someone  without  a political  background  read  and  comprehend  an  article  about  the  election. 

I found  most  of  my  sources  in  Joyner  Library  on  the  ECU  campus,  but  many,  if  not  all,  of 
these  sources  can  be  found  at  any  public  or  academic  library. . . In  the  future  I might  change 
the  pathfinder  by  adding  a section  aimed  at  school-age  children  or  by  re-writing  it  with  that 
audience  in  mind. . . .If  I added  this  section  to  the  pathfinder  or  re-wrote  the  whole  thing,  I 
would  actually  strengthen  my  mastery  of  Objective  3 because  I would  be  helping  answer  even 
more  reference  questions  using  print  and  electronic  sources.  . . This  project  was  a valuable 
one  for  me  because  I will  be  creating  pathfinders  often  as  a school  media  specialist.  Overall, 
the  project  was  very  time  consuming,  but  in  the  end,  I was  able  to  acquire  or  hone  the  above- 
mentioned  skills  for  future  use  in  my  professional  field. 
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There  is  one  disadvantage  for  designating  a specific  project  for  the  portfolio.  Students  may  devote  an 
unbalanced  amount  of  time  and  energy  to  the  identified  project  and  neglect  other  equally  important 
assignments/projects.  See  this  student's  comment  and  how  she  has  spent  the  majority  of  her  time  on  designated 
projects  - 

I initially  thought  that  we  would  undergo  several  projects  in  each  class  and  we  would  choose 
our  best  to  be  presented  in  our  portfolio.  However  after  these  last  two  semesters,  I see  there’s 
only  one  main  project  per  course  so  now  I contribute  more  planning  time  to  that  one  project 
b/c  I know  it  will  be  presented  in  the  portfolio  and  it  needs  to  be  my  best  work. 

This  student  unwittingly  admitted  extra  time  for  the  portfolio  project  as  she  responded  to  the  survey  question  on 
"time  management". 


Inter- rater  Reliability 

A major  challenge  was  bringing  faculty  to  consensus  on  collection  methods  and  format  for  data  used  as 
evidence  in  the  portfolio.  Initially,  faculty  envisioned  the  evaluation  process  as  a duplication  of  efforts.  "Why 
should  I evaluate  student  products  if  they've  already  been  evaluated  and  graded?"  was  the  general  response. 

With  time,  faculty  began  to  see  the  evaluation  of  the  reflective  writings  as  one  valid  measure  of  students' 
mastery  of  key  concepts  and  skills  in  the  courses. 

Simple  logistics  such  as  how  and  when  to  establish  evaluation  committees  also  presented  obstacles. 
Rubrics  had  to  be  developed  with  clear  indicators  that  were  directly  aligned  with  either  the  Program  Objectives 
or  standards  and  competencies  recommended  by  state  agencies  and  learned  societies.  Much  deliberation  was 
required  before  all  faculty  were  in  agreement.  Once  this  was  accomplished,  committees  were  formed  and 
student  portfolios  were  efficiently  and  reliably  evaluated  at  the  end  of  each  semester.  Students  were  notified  of 
deadlines  well  in  advance  of  due  dates.  This  allowed  faculty  adequate  time  for  the  evaluation  process  and 
provided  the  student  time  to  make  necessary  revisions  or  updates  to  their  final  product. 

Each  portfolio  was  evaluated  by  at  least  three  faculty  members.  Each  indicator  within  the  rubric 
included  a scale  of  0 to  5.  Total  points  possible  for  each  indicator  being  5 and  total  points  for  the  entire  portfolio 
being  equal  to  the  number  of  indicators  for  the  entire  rubric.  The  expectation  was  that  100%  of  the  students 
reached  mastery  following  recommendations  for  revisions. 

Recommendations 

Two  advanced  degree  programs  at  East  Carolina  University  have  used  electronic  portfolios  as  the 
culminating  project  before  graduation.  The  Master  of  Library  Science  began  using  the  electronic  portfolio  in 
2001  (Brown  & Boltz,  2002),  and  more  recently,  the  Master  of  Education  in  Instructional  Technology  requires  a 
similar  product  during  the  student's  final  semester.  Upon  entry  into  an  advanced  degree  program,  students 
usually  must  enroll  in  one  or  more  introductory  cours  es  related  to  their  profession.  Seminars  or  modules 
devoted  to  the  development  of  electronic  files  and  use  of  public  domain  file  transfer  software  (e.g.  WSftp)  will 
prepare  students  to  master  the  basic  technical  skills  needed  for  their  portfolio.  Use  of  a dedicated,  and  password 
protected,  server  is  recommended  for  maintaining  the  database  with  all  student  portfolios.  Access  should  be 
restricted  to  the  student's  personal  directory  and  limited  to  faculty  teaching  in  the  program  area.  A screen  shot  of 
the  database  is  displayed  in  Figure  1.  Only  faculty  with  administrator  passwords  can  view  the  entire  database. 
Students  may  provide  prospective  employers  with  the  URL  to  their  portfolio  homepage  and  these  can  be  viewed 
by  any  browser. 
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Figure  1 . First  screen  of  database  with  all  student  portfolios. 


This  ensures  privacy  for  students  and  allows  a systematic  method  for  indexing  student  products  that 
can  be  used  for  evidences  needed  for  accreditation  and  program  approval  by  state  departments  of  education. 

The  dedicated  server  eliminates  the  necessity  for  storing  student  work  samples,  semester  after  semester,  on 
faculty  hard  drives  and  file  cabinets.  In  addition,  students  may  use  the  URL  to  their  directory  for  personal  job 
searches  and  promotions.  The  program  areas  described  in  this  paper  allows  student  access  to  their  personal 
directory  and  files  for  a time  period  of  up  to  three  years  after  graduation.  This  provides  an  incentive  for  students 
to  invest  additional  energy  and  creativity  into  the  quality  of  their  artifacts  and  design  of  the  portfolio. 

Faculty  from  these  programs  report  favorable  responses  from  students.  Students  have  adopted  the  use 
of  portfolios  as  a method  for  reflectively  reporting  their  personal  reaction  to  course  projects  and  assignments. 
Support  for  these  conclusions  come  from  (1)  anecdotal  comments  from  students  and  their  advisors,  (2)  written 
responses  to  end  of  course  surveys,  and  (3)  reflective  writings  that  justify  student  artifacts  as  representative  of 
standards  and  competencies.  Students  have  become  increasingly  more  sophisticated  in  their  knowledge  and 
understanding  in  the  portfolio  as  a tool  for  authentic  assessment.  Although,  many  professional  programs 
include  the  “showcase”  portfolio  as  part  of  the  student’s  culminating  experience,  a higher  purpose  for  the  use  of 
portfolios  is  its  use  as  a tool  for  constructed  learning  (Paulson  & Paulson,  1994).  The  student  must  be  able  to 
articulate  how  his  or  her  products  reflect  the  criteria  established  by  the  standard.  This  facilitates  transfer  to 
actual  working  situations  as  the  student  enters  his  or  her  initial  professional  setting. 

As  the  instructor/assessor  guides  and  provides  council  during  the  development  of  the  contents  of  a 
student’s  portfolio,  there  is  opportunity  for  dialog  and  exchange  of  ideas.  The  student  is  able  to  see  the 
modeling  of  professional  behaviors,  attitudes,  and  skills  from  a closer  perspective  than  the  usual  interactions 
within  the  classroom  setting.  This  is  particularly  important  advantage  for  virtual  classes  and  online  degree 
programs.  Indeed,  the  continual  evaluation  of  assignments,  and  how  these  relate  to  professional  standards, 
affirms  the  student’s  professional  goals,  or  in  some  cases,  leads  to  consideration  for  a change  in  career  paths. 

For  the  instructors  and  faculty  of  the  program,  there  is  opportunity  for  collaboration  with  colleagues  to  examine 
and  evaluate  program  goals  and  objectives.  Individual  evaluation  of  program  objectives  and  how  these  are 
reflected  in  course  syllabi,  activities,  and  assignments  are  a natural  product  of  the  process  portfolio. 

There  is  growing  evidence  that  portfolio  assessment  is  a valid  measure  of  skill  and  concept  attainment, 
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and  that  there  is  reliability  of  measurement  for  predicting  student  achievement  following  graduation.  However, 
research  in  this  area  of  assessment  is  still  limited.  It  is  difficult  to  obtain  data.  There  are  misconceptions  about 
the  purpose  and  functions  for  portfolios,  and  authentic  assessment  requires  a large  investment  of  time. 
Additional  time  is  needed  for  training  of  assessors  and  for  counseling  students.  From  this  review  and  from  my 
own  observation  of  the  portfolio  process,  the  additional  time  needed  is  outweighed  by  benefits  for  student 
learning  and  for  program  improvements.  The  issues  for  validity  and  reliability  should  be  considered  before 
implementation.  Faculty  should  clearly  define  any  or  all  of  the  following:  program  objectives,  national  or 
professional  standards  and  competencies,  and  performance  indicators  that  represent  the  standards.  These  should 
be  comrnu  nicated  to  the  student  when  entering  the  program.  In  addition,  students  should  be  advised  and 
mentored  with  regard  to  quality  of  portfolio  contents  and  how  these  reflect  the  specified  standards  and 
objectives.  Finally,  assessors  should  be  trained  in  both  consensus  scoring  and  independent  scoring  procedures, 
and  in  determining  a holistic  evaluation  of  the  final  product. 
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Introduction 

This  paper  aims  to  identify  and  to  understand  the  role  of  the  logical,  the  affective  and  the  ethical 
dimensions  of  knowledge  in  the  online  interactions  of  post-secondary  teachers  and  students.  By  understanding 
how  these  dimensions  are  interwoven,  I intend  to  demonstrate  that  the  instructor’s  course  design  and  teaching 
strategies  must  take  them  into  consideration  in  order  to  achieve  a healthy  learning  environment.  By  healthy 
learning  environment  I mean  the  online  construction  of  a knowledge  building  community  in  which  the  students 
can  develop  arguments  in  order  to  acquire  a more  profound  understanding  of  the  topics  and  develop  caring  and 
fulfilling  relationships  in  a respectful  atmosphere.  I adopt  a constructivist  perspective  based  on  genetic 
epistemology  (Piaget,  1950,  1991)  as  well  as  contributions  by  socio -constructivism  (Vygotsky,  1979)  and 
modern  cognitive  science  distributed  approaches  (Salomon,  1993).  Methodologically,  I integrate  two  research 
methods:  discourse  analysis  and  the  case  study.  Data  collected  come  from  transcripts  of  electronic  conferencing, 
course  documents  and  interviews.  The  study  suggests  that  online  interactions  are  shape  by  the  teaching  strategy 
adopted  by  the  instructor  in  the  electronic  conferences. 

Theoretical  framework 

This  paper  aims  to  identify  and  to  understand  the  role  of  the  logical,  the  affective  and  the  ethical 
dimensions  of  knowledge  in  the  online  interactions  among  post-secondary  teachers  and  students.  According  to 
Piaget,  social  knowledge  has  three  interconnected  dimensions:  (1)  the  rules  expressed  by  logical  operations,  (2) 
the  values  attributed  to  meanings  that  are  of  two  orders:  affective  and  ethical,  and  (3)  the  signs  used  for 
communication  (Piaget,  1976).  Both  logical  systems  and  meaning  systems  are  expressed  by  signs  coded  through 
language.  In  the  case  of  “online  language”  signs  are  verbal  and  non-verbal. 

I apply  the  Piagetian  Grize’s  communication  model  (1991)  with  a view  of  understanding  the  symbolic 
ecology  of  humans  as  an  open  system.  This  model  is  consistent  with  the  understanding  of  Habermas,  according 
to  whom  social  theory  can  not  ignore  the  dimension  of  discourse  (1987).  This  symbolic  ecological  approach  is 
interested  in  the  psychogenesis  as  well  as  the  sociogenesis  of  communication,  integrating  genetics  and  history 
whenever  it  is  possible.  In  this  paper,  I provide  examples  of  two  university  mixed-mode  courses  in  which 
electronic  conferences  were  integrated.  By  studying  the  communication  dynamics  of  the  symbolic  ecological 
environment  in  which  the  actions  of  the  professors  adopting  a given  teaching  made  possible  certain  kinds  of 
interaction  with  and  among  the  students,  I aim  to  identify  and  to  understand  the  role  of  logics,  affectivity  and 
values.  Consistent  with  genetic  epistemology,  these  three  interconnected  dimensions  of  knowledge  could 
provide  an  understanding  on  the  dynamics  of  configurations  of  meanings  (Campos,  1998).  This  understanding 
might  shed  a light  on  the  complexity  of  communication  and  its  role  on  the  learning  processes  of  the  students. 

Methodology  and  data  sources 

The  core  of  our  multi-method  approach  is  discourse  analysis  of  electronic  conferencing  and  the  case 
study.  Our  discourse  analysis  method  is  synchronic  as  well  as  diachronic.  Synchronically,  it  highlights  the 
logical  forms  and  applied  meanings  present  in  the  equilibration  process  of  conceptual  assimilation  and 
accommodation  that  leads  to  the  construction  of  arguments.  Diachronically,  it  draws  the  relationships  among 
configurations  of  meanings  that  form  the  online  discourse  that  is  expressed  by  meaning  implication  (Piaget, 

1991;  Campos,  1998,  2000).  Further  information  on  the  method  can  be  found  in  Campos,  2004. 

Data  sources  and  subjects 

Data  came  from  two  postsecondary  courses: 

a)  A Chemistry  Lab  course 

b)  A Medicine  course  on  urology  and  the  kidney 

Both  courses  were  mixed-mode  (face-to-face  and  online),  built  around  an  online  learning  environment  offering 
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many  tools  such  as  access  to  online  materials,  quizzes,  self-assessment  of  the  progression  of  learning, 
evaluation  tools,  private  internal  e-mail  as  well  as  a number  of  online  discussion  conferences  dedicated  to 
different  topics  (WebCT).  We  collected  transcript  data  from  electronic  conferences  of  both  courses,  and  made 
interviews  with  both  professors  and  students. 

Results 

Chemistry  Laboratory  This  chemistry  mixed-mode  (face-to-face  and  online)  course  was  conceived  for 
students  registered  in  science  program.  In  the  studied  session  (fall  2003),  the  chemistry  laboratory  had  133 
registered  students.  The  goal  was  that  of  introducing  students  to  experimental  chemistry  by  adopting  an 
approach  integrating  analytical,  organic  and  physical  chemistry.  This  one  session  course  was  organized  around 
some  lectures  followed  by  lab  sessions  in  which  the  students  were  requested  to  perform  a number  of 
experiences.  The  online  component  was  electronic  conferencing.  The  evaluation  comprised  two  elements: 
laboratory  activities  and  a final  exam.  The  laboratory  elements  of  the  evaluation  were  the  quality  of  lab  work 
(10%)  as  well  as  lab  reports  (55%).  The  final  exam  counted  35%  of  the  mark.  Online  conferencing  was 
evaluated  in  terms  of  participation  (2%)  and  quality  (3%):  these  points  were  comprised  within  the  laboratory 
elements  of  evaluation.  For  the  participation  mark,  the  professor  used  the  conferencing  statistical  tool  that 
shows  how  many  times  a student  wrote  and  read  messages. 

The  professor  was  not  a first-timer:  he  already  taught  this  course  a number  of  times.  This  session  was 
the  second  in  which  he  integrated  electronic  conferencing  in  this  course.  This  professor  also  integrates  online 
conferencing  in  other  courses,  meaning  that  this  second  experience  in  the  Chemistry  Laboratory  course  was  not 
his  second  experience  with  online  conferencing.  According  to  this  professor,  online  conferencing  was  used 
because  of  its  efficacy.  First  of  all,  the  tool  was  considered  to  be  useful  because  students  could  ask  questions 
about  scientific  concepts  that  they  did  not  understand.  Secondly,  the  professor  considered  essential  the  mastery 
of  the  language  to  communicate  precisely  the  intended  meanings  and  conferencing  was  seen  as  a good  writing 
exercise.  Thirdly,  electronic  conferencing  was  considered  to  be  a collaborative  tool  in  which  students  could  help 
each  other.  The  professor  decided  neither  to  interfere  constantly  nor  to  answer  or  comment  immediately  after  a 
question  or  comment  was  published.  The  strategy  adopted  was  that  of  waiting  to  see  whether  the  students  would 
try  to  answer  and  help  their  colleagues  before  any  intervention  by  the  professor  was  needed.  According  to  the 
professor,  intervening  happened  when  the  answer  already  given  by  a student  was  incomplete  or  incorrect,  or 
when  nobody  had  answered  a question,  or  else,  to  provide  information  about  the  course.  The  professor  finds 
electronic  conferencing  to  be  adequate  for  this  kind  of  course  because  even  if  the  students  demonstrate  to  be 
capable  of  solving  chemistry  problems,  they  normally  are  unable  to  properly  explain  what  they  do  and  why.  The 
availability  of  such  a communication  tool  allows  them  to  explain  to  each  other  the  processes  involved  in  the 
calculations  which  would  enable  learning.  The  professor  also  points  out  that  the  dynamics  enables  first  year 
students  to  do  things  that  only  last  year  students  used  to  do  beforehand.  The  students  that  we  interviewed  found 
participation  in  the  online  conferences  a rich  and  useful  experience.  They  considered  that  the  possibility  of 
asking  questions  online  helped  them  to  follow  the  course.  They  also  found  that  this  tool  was  particularly 
relevant  for  those  who  are  shy  and  have  a difficult  time  asking  questions  in  the  classroom.  The  students 
perceived  the  strategy  adopted  by  the  professor  in  the  intended  way.  According  to  them,  the  professor’s  strategy 
was  one  of  trying  not  to  give  a response  right  away  to  encourage  discussion.  However,  they  were  not  in 
agreement  concerning  the  character  of  the  community:  one  student  considered  it  a learning  community,  while 
the  other  a “consulting”  community,  the  students  did  not  share  the  same  degree  of  enthusiasm  about  the 
experience  although  they  considered  it  to  be  positive. 

Discourse  analysis  We  start  with  a quantitative  view  of  the  studied  conference  to  introduce  the  reader 
to  the  major  trends  of  interaction.  The  study  was  done  upon  45  messages  of  a general  conference,  open  to 
everyone.  Participation  was  not  mandatory.  Messages  with  and  without  arguments  were  split  almost  equally 
(Figure  1). 


Messages 

# 

% 

With  arguments 

23 

51 

Without  arguments 

22 

49 

Figure  1 - Presence  of  arguments 


However,  only  a third  of  the  total  of  messages  with  arguments  had  a construction  (i.e.  somebody  building  upon 
what  another  person  had  written).  A significant  number  of  messages  with  arguments  presented  no  construction 
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at  all,  meaning  that  what  those  persons  wrote  died  right  there,  without  a continuation.  It  is  expected  that  most 
messages  without  arguments  presented  no  construction  at  all  but  it  is  interestingly  enough  to  see  that  some 
messages  presented  a sort  of  construction  (meaning)  anyhow  (Figure  2). 


Messages 

# 

% 

With  arguments  and  construction 

14 

31 

With  arguments  and  without  construction 

9 

20 

Without  arguments  and  construction 

3 

7 

Without  arguments  and  without  construction 

19 

42 

Figure  2 - Arguments  and  construction 


In  terms  of  values,  most  messages  had  a positive  content.  However,  an  almost  even  number  of  messages  were 
negative  or  neutral.  This  trend  suggests  that  this  conference  was  not  particularly  engaging  (Figure  3). 


Values  1 

# 

% 

Positives  + 

18 

40 

Negatives  - 

14 

31 

Neutral  +- 

13 

29 

Figure  3 - Intensity  of  values 


When  relating  the  kind  of  message  with  the  value  intensity  attributed  to  it,  we  verify  that  most  messages  with 
arguments  had  a negative  content,  followed  by  a reasonable  number  of  positive  and  neutral  ones.  We  also  note 
that  messages  without  arguments  were  mainly  positive  but  with  a fair  number  of  neutral  and  negative  ones 
(Figure  4). 


Values  1(%) 

Positives  + 

Negatives  - 

Neutral  +- 

With  arguments  and  construction 

11 

11 

9 

With  arguments  and  without  construction 

7 

9 

4 

Without  arguments  and  construction 

2,5 

0 

2,5 

Without  arguments  and  without  construction 

20 

11 

13 

Figure  4 - Intensity  of  values  according  to  messages  with  or  without  arguments,  with  or  without  construction 


In  terms  of  the  content  of  messages,  most  messages  had  an  affective  value  attached  to  the  writing.  A non 
negligible  number  was  just  information  publishing.  A fair  number  of  messages  had  a moral  content  (Figure  5). 


Values  2 

# 

% 

Affective 

28 

62 

Moral 

7 

16 

Informational  (neutral) 

10 

22 

Figure  5 - Type  of  value 


Concerning  how  the  type  of  value  was  distributed,  it  is  worth  noting  that  all  messages  with  a moral  content  were 
part  of  a discussion  in  which  participants  constructed  upon  each  others’  comments.  Most  messages  with  an 
affective  content  were  found  in  messages  without  arguments  or  construction,  i.e.  manifestations  of  care  that 
were  not  responded  to  (Figure  6). 


Values  2 (%) 

Affective 

Moral 

Neutral 

With  arguments  and  construction 

11 

13 

7 

With  arguments  and  without  construction 

13 

2 

5 

Without  arguments  and  construction 

5 

0 

0 

Without  arguments  and  without  construction 

33 

0 

11 

Figure  6 - Type  of  values  according  to  messages  with  or  without  arguments,  with  or  without  construction 


When  we  cross  the  values,  what  is  striking  is  that  most  messages  with  a moral  content  were  negative.  In 
addition,  affectivity  does  not  mean  necessarily  to  be  positive  as  show  in  the  table  (Figure  7). 


160 


Crossing  values  (%) 

Affective 

Moral 

Neutral 

Positives  + 

36 

4 

0 

Negatives  - 

20 

11 

0 

Neutral  +- 

0 

7 

22 

Figure  7 - Crossing  intensity  and  type  of  values 


We  would  like  to  provide  additional  quantitative  information  that  is  interesting  analyzing.  From  those  messages 
with  arguments  and  construction  (14  messages),  8 were  built  upon  previous  questions,  2 upon  themes  unrelated 
to  any  arguments,  and  only  4 upon  a previous  argument.  Three  of  these  four  messages  had  a moral  content.  In 
addition,  we  verify  a phenomenon  of  over  presence:  57.8%  of  the  messages  (26)  were  written  by  the  professor  - 
56.5%  (13)  of  the  total  number  of  messages  with  arguments  (23)  - most  of  them  to  answer  questions.  Hereunder 
we  show  examples  of  such  a construction  dealing  with  moral  dilemma: 

Message  n (Informational  / Neutral  value  - with  argument  - no  construction) 

(Professor) 

Premise  - I must  remember  you  that  you  should  a copy  of  your  laboratory  report  with  your  assignment... 

Premise  2 + conclusion  - IF  you  don’t  do  this  (THEN)  your  evaluation  will  be  affected. . . Good  work.  (Signed 
with  initials) 

Message  n’  (Negative  / Moral  value  - with  argument  - with  construction  upon  the  previous  theme) 

(Student  A) 

Premise  1 - I need  to  express  my  deception  concerning  the  delivery  of  assignments. 

Premise  2 - As  other  students  I left  my  assignment  in  the  requested  box  left  is  a corridor  without  surveillance. 
Premise  3 - Anyone  could  steal  or  copy  the  content  of  one  or  many  of  those  assignments. 

Premise  4 - Some  other  things  that  happened  in  this  course  make  me  doubt  of  the  reliability  of  our  colleagues. 2 
Premise  5 - IF  this  method  continues,  THEN  is  because  nothing  serious  had  already  happened... 

Premise  7 - This  is  another  reason  not  to  take  a chance . . . 

Conclusion  - IF  (the  previous  are  true)  THEN  it  would  be  better  to  assure  security... 

Message  n”  (Negative  / Moral  value  - with  argument  - with  construction  upon  the  previous  argument) 

(Student  B) 

Premise  1 - 1 agree.  (Given  what  you  said,  THEN)  I would  also  prefer  to  give  my  assignment  to  the  professor... 
Premise  2 - We  worked  hard  for  this. . . and  it  would  be  a pity  to  have  someone  punished  by  a stealer. 

Conclusion  - (IF  this  continues  THEN)  chances  might  be  low  but  previous  happenings. . . in  this  course. . . do 
not  assure  anybody. 

Message  n’”  (Positive  / Moral  value  - with  argument  - with  construction  upon  the  previous  argument) 
(Professor) 

Premise  1 - I will  investigate  (the  previous  cases  reported). 

Premise  2-1  will  take  measures  in  the  future  to  avoid  risks  of  fraudulent  behaviour. 

Premise  3 - As  I remember,  in  a previous  case  similar  to  the  one  that  you  reported,  the  student  was  expelled 
from  the  university. 

Premise  4 - It’s  certain,  though,  that  nobody  saw  him. . . 

Premise  2 + conclusion  - IF  you  have  precise  facts  (THEN),  please,  let  me  know...  (Signed  with  initials) 

In  order  to  also  present  one  of  the  few  examples  of  knowledge  co-construction  (of  arguments)  among  students, 
we  chose  one  of  the  longest  in  which  the  students  help  each  other  in  the  search  of  a response  to  a difficult 
problem: 

Message  m (Negative  / Affective  value  - with  argument  - no  construction) 

(Student  A’) 

Premise  1 - The  table  at  page  4 (in  the  college  book)  that  we  must  fulfill  does  not  respect  the  rules  concerning 
significant  numbers  particularly  in  the  mmol  column. . . 

Premise  2 - We  perform  a three  rule  and  the  equivalences  are  expressed  with  2 numbers... 
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Conclusion  1 - THEN,  the  data  that  should  be  written  (in  the  college  book)  should  also  have  2CS. 

Conclusion  2 - (IF)  One  could  not  be  more  precise  than  the  least  precise  data.  THEN,  why  it  is  written  3CS  in 

the  correction  (of  the  exercise)? 

Message  m’  (Positive  / Moral  value  - with  argument  - with  construction  upon  the  previous  theme) 

(Student  B') 

Premise  1 - I also  noted  that  it  seems  that  the  rule  was  not  respected. . . 

Premise  2 - In  addition. . . I do  not  understand  why  the  uncertainty  value  of  0.05ml  is  multiplied  by  2. . . From 

where  this  information  come  from???  Thanks  in  advance 
Conclusion  1 - (Given  this)  THEN,  an  explanation  is  needed  to  avoid  problems  in  the  exam. 

Message  m”  (Informational  / Affective  value  - with  argument  - with  construction  upon  the  previous  question) 
(Student  C’) 

Premise  1 - Hi,  IF  there  are  two  additive  measures  measured  with  the  same  instrument,  THEN  the  respective 

absolute  uncertainty  values  should  be  summed  up. 

Premise  2 - For  example.. . IF  you  want  to  measure  10.00ml... 

Conclusion  1 - (Given  this)  THEN,  the  result  is  a volume  of  10.00  +-  0.02ml. 

Medicine  Course  This  mixed-mode  (face-to-face  and  online)  course  on  urology  and  the  kidney  was 
conceived  for  students  registered  in  the  medicine  program.  In  the  studied  session  (fall  2003),  the  chemistry 
laboratory  had  123  registered  students.  The  goal  was  that  of  introducing  students  to  understanding  the 
functioning  of  the  urological  system  having  kidneys  at  its  core.  The  course  had  the  help  of  24  tutors  (15 
nephrologists,  5 urologists,  3 residents  in  nephrology  and  1 generalist).  In  addition,  it  had  4 non  expert  teaching 
assistants.  The  professor  coordinated  this  team  during  the  session.  This  one  session  course  was  organized 
around  lectures,  working  groups  focusing  on  problem-solving  processes,  modules  of  self-learning,  modules  of 
self-assessment,  online  conferencing,  and  other  materials.  The  course  had  strong  online  components.  In  addition 
to  online  materials  (course  notes,  Power  Point  presentations,  a calendar  and  a private  e-mail  system),  sessions  of 
self-learning  were  prepared  to  follow  discussions  of  the  problem-solving  working  groups.  The  students  could 
repeat  the  online  modules  until  mastering  the  content.  The  self-assessment  tool  had  a similar  goal  but  could  only 
be  used  once.  Electronic  conferencing  was  used  as  an  online  peripheral  complement  of  the  face-to-face  working 
groups.  The  evaluation  comprised  three  elements:  final  exam  (75%),  evaluation  of  the  student  by  a mentor 
(20%)  and  participation  in  the  online  learning  environment  (5%).  The  last  item  was  broke  down  in  2%  for 
participation  in  the  online  conferences,  2%  for  answering  the  self-assessment  quiz,  and  1%  for  answering  an 
online  questionnaire  about  the  learning  environment. 

We  needed  to  explore  the  mechanics  of  the  problem-solving  working  groups  to  understand  how  the 
conferences  were  used.  The  problem-solving  working  groups  had,  each  one,  a dozen  students  tutored  by  an 
expert.  There  were  seven  problems  to  be  solved  and  hence  seven  online  conferences  were  created.  Given  the 
high  number  of  students  in  the  course,  for  each  problem  (and  related  online  conference)  there  were  a number  of 
assigned  groups.  These  groups  had  as  goal  a problem-solving  assignment  that  was  structured  in  the  course  notes 
published  in  the  online  learning  environment  used  in  the  course.  The  first  task  of  the  group  was  one  of 
clarifying  what  the  objectives  of  the  discussions  were  so  as  to  enable  the  resolution  of  the  discussed  problem. 

All  this  process  was  followed  by  a tutor  and  had  a working  group  student  assigned  to  lead  the  process.  Parallel 
to  these  face-to-face  discussions,  the  students  were  requested  to  participate  at  the  online  conferences  open  for 
each  of  the  seven  problems.  The  students  were  requested  to  participate  in  those  conferences  in  order  to  clarify 
and  define  the  objectives  specific  to  a given  problem.  Other  online  conferences  were  open  (for  discussion,  for 
instance)  but  did  not  work.  The  only  online  conferences  that  worked  were  those  related  to  the  problem-solving 
working  groups.  However,  most  students  just  “listed”  the  requested  objectives  with  no  discussion  and  sense  of 
politeness.  Although  the  professor  had  presented  a netiquette  link,  the  resulting  discussions  could  hardly  be 
called  a “conversation”,  so  unattached  were  they. 

There  are  already  two  years  that  the  professor  is  using  electronic  conferencing  in  this  medicine  course. 
The  goal,  according  to  the  professor,  was  to  make  homogeneous  the  objectives  of  the  problem-solving  working 
groups  that  were  discussing  the  same  problem.  In  order  to  help  them  to  fix  the  objectives,  the  professor  found 
that  electronic  conferencing  would  be  ideal.  Although  the  professor  believes  that  students  did  not  like  electronic 
conferencing,  there  are  doubts  about  whether  it  was  really  useful.  The  intervention  strategy  was  one  of  waiting 
for  student -student  interaction  and  answering  questions  in  the  hope  that  conferencing  would  create  a need  of 
mutual  help.  However,  the  professor  recognizes  that  the  online  conference  strategy  adopted  to  facilitate  logistics 
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provided,  in  fact,  minimum  effort  from  students.  The  professor  had  the  impression  that  students  only 
participated  in  order  to  get  points  (which  were  considered  significant  in  the  context  of  a medicine  course)  and 
was  disappointed  with  what  was  considered  to  be  poor  results.  The  students  interviewed  found  the  online 
experience  positive  but  in  their  answers  they  focused  mostly  on  other  aspects  of  the  online  experience  rather 
than  on  electronic  conferencing.  It  was  noted  that  although  participation  was  mandatory  the  use  of  conferences 
had  a positive  inpact  on  the  process. 

Discourse  analysis 

The  study  was  done  upon  49  messages  of  7 problem-solving  working  group  online  conferences,  open  to  those 
assigned  groups.  Participation  was  not  mandatory  but  students  got  participation  points.  Messages  with 
arguments  were  almost  non  existent:  most  messages  did  not  have  any  arguments  (Figure  8). 


Messages 

# 

% 

With  arguments 

3 

6 

Without  arguments 

46 

94 

Figure  8 - Presence  of  arguments 


Co-construction  of  arguments  was  also  almost  non-existent  but  some  construction  could  be  seen  in  messages 
without  arguments,  mostly  because  of  the  use  of  the  reply  function  to  answer  a question  (Figure  9).  The 
professor  participated  in  26.5%  of  the  conference  (13  messages  written)  and  was  responsible  for  84.6%  of 
messages  with  construction  (9  messages).  Most  messages  had  no  arguments  and  construction:  just  information 
publication  (“listing  the  objectives”). 


Messages 

# 

% 

With  arguments  and  construction 

2 

4 

With  arguments  and  without  construction 

1 

2 

Without  arguments  and  construction 

9 

18 

Without  arguments  and  without  construction 

37 

76 

Figure  9 - Arguments  and  construction 


In  terms  of  values,  most  messages  had  a negative  or  neutral  content.  Most  students  did  not  even  use  to  say 
“Hello”  (Figure  10). 


Values  1 

# 

% 

Positives  + 

15 

31 

Negatives  - 

24 

49 

Neutral  +- 

10 

20 

Figure  10-  Intensity  of  values 


As  stated  above,  most  negative  and  neutral  messages  were  those  that  just  presented  crude  information  (the 
objectives)  with  any  kind  of  attempt  to  interact.  Many  positive  messages,  though,  were  those  that  presented 
arguments  (Figure  11). 


Values  1 (%) 

Positives  + 

Negatives  - 

Neutral  +- 

With  arguments  and  construction 

2 

0 

2 

With  arguments  and  without  construction 

0 

0 

2 

Without  arguments  and  construction 

12 

4 

2 

Without  arguments  and  without  construction 

16 

45 

15 

Figure  11-  Intensity  of  values  according  to  messages  with  or  without  arguments,  with  or  without  construction 


As  expected,  most  messages  had  a neutral  information  value  (Figure  12).  A significant  number  had  also  an 
affective  value  and  few  messages  a moral  value  (all  of  them  written  by  the  professor  exercising  authority  in  a 
way  or  another). 
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Values  2 

# 

% 

Affective 

21 

43 

Moral 

3 

6 

Informational  (neutral) 

25 

51 

Figure  12  - Type  of  value 


Concerning  how  the  types  of  value  were  distributed,  the  figure  below  ( 1 3)  j ust  highlights  what  has  already  been 
suggested  above. 


Values  2 (%) 

Affective 

Moral 

Neutral 

With  arguments  and  construction 

4 

0 

0 

With  arguments  and  without  construction 

2 

0 

0 

Without  arguments  and  constmction 

12 

4 

2 

Without  arguments  and  without  construction 

25 

2 

49 

Figure  13  - Type  of  values  according  to  messages  with  or  without  arguments,  with  or  without  construction 


When  we  cross  the  values,  we  further  confirm  the  trends  of  these  conferences.  Negativity  was  related  to  neutral 
(informational)  statements  because  most  students  just  published  objectives  disregarding  any  usual  polite  words 
used  in  written  communication  such  as  e-mail  (Figure  14). 


Crossing  values  (%) 

Affective 

Moral 

Neutral 

Positives  + 

31 

0 

0 

Negatives  - 

0 

4 

45 

Neutral  +- 

12 

2 

6 

Figure  14  - Crossing  intensity  and  type  of  values 


Hereunder  we  show  one  of  the  few  examples  of  knowledge  construction  emerging  from  the  conferences  in 
which  two  students  and  the  professor  discussed  a problem  to  be  solved: 

Message  p (Informational  / Affective  value  - with  argument  - no  construction) 

(Student  A”) 

Premise  1 - Hi  Dr....  In  the  self-assessment  quiz  of  the  problem-solving  working  group  6,  question  8,  response 
about  hypocalcaemia  is  related  to  the  augmenting  of  phosphor-calcium  product  and  vitamin  D. . . 
Premise  2-1  thought  I had  understood  from  my  readings  that  the  phosphor-calcium  product  was  a kind  of 
constant,  THEN  how  could  it  augment? 

Conclusion  - (IF  the  previous  THEN)  I understand  that  as  the  augmentation  of  the  phosphate  triggering  a 

diminution  of  calcium  to  guarantee  that  the  phosphor-calcium  product  does  not  change. . . (Signed) 

Message  p’  (Informational  / Affective  value  - with  argument  - with  construction  upon  the  previous  argument) 
(Student  B”) 

Premise  1 - Hi  there,  I am  not  sure  whether  I have  the  good  answer  or  not  but  if  P04  augments,  it  produces  the 
diminution  of  Ca2+  because  it  will  precipitate  both  the  P04  and  the  Ca...  Conclusion  - (IF  this) 

THEN  it  will  lead  the  phosphor-calcium  product  at  a constant  level  (after  having  diminished  the 
Ca2+). 

Message  p”  (Positive  / Affective  value  - no  argument  - with  construction  upon  the  previous  message) 
(Professor) 

Rhetorical  procedure  - You  have  the  good  answer!  (Signed) 

Conclusion 

Our  research  suggests  that,  although  the  quality  of  online  interaction  cannot  be  strictly  attributed  to  a 
given  teaching  strategy,  the  nature  of  the  discipline,  planning  of  the  course  and  management  of  the  online 
conferences  shape  the  level  of  participation.  The  use  of  online  conferencing  can  hardly  be  considered  a success 
in  these  courses  but  our  goal  here  was  not  one  of  evaluating  them  in  terms  of  a dichotomy  success/  failure.  In 
fact,  the  conferences  served  those  who  asked  questions  and  had  them  answered,  served  as  a communication  tool 


164 


about  administration  information  the  professors  were  willing  to  broadcast,  and  served  for  those  unhappy  voices 
that  used  the  conferences  either  for  legitimate  or  non  legitimate  reasons.  Argument  co-construction  was 
reasonable  in  the  Chemistry  course  and  poor  in  the  Medicine  course  indicating  a low  to  medium  level  of  online 
learning  emerging  from  the  written  interactions.  However,  nothing  could  be  said  about  effective  student 
learning  in  those  courses  because  they  were  not  limited  to  the  virtual  and  also  had  a face-to-face  component. 

These  experiences,  however,  show  the  potential  of  the  use  of  online  conferencing.  They  certainly 
should  be  re-thought  in  the  context  of  the  sciences,  and  the  strategies  employed  adjusted  with  a goal  of 
extracting  a more  consistent  engagement  and  participation.  This  adjustment  should  be  one  of  reconsidering  the 
rules  employed  in  order  to  avoid  negativity  in  the  exchange  of  values,  critical  for  promoting  co-argument 
building  and  knowledge  construction. 
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Abstract 

This  study  investigated  the  potential  influence  of  students’  personalities  on  online 
discussion,  group  interaction,  task  engagement,  and  communication  duration.  The  results  of  this 
study  indicate  that  personality  affects  communication  type,  pattern  and  duration;  productive  group 
structures  result  in  sustained  discussion.  The  results  also  showed  that  students’  feelings,  attitudes  and 
opinions  about  their  online  discussion  experiences  are  positive.  To  optimize  effective  online 
communication,  grouping  strategies  should  combine  personality  profiles  when  designing  online 
collaborative  learning  activities. 


Introduction 

The  value  of  collaborative  learning  experiences  has  been  well  established  (Campbell  & Smith,  1997; 
Johnson,  & Johnson,  1989;  Johnson,  Johnson,  & Smith,  1997,  1999).  Online  collaborative  learning  has 
typically  been  studied  within  the  context  of  learning  communities  (Sherry,  1996;  Sharon  & Edward,  1999; 
Solloway  & Harris,  1999).  Though  communities  are  composed  of  groups  of  people  with  common  interests  (The 
American  Heritage  Dictionary,  1997),  individuals  within  communities  bring  unique  personalities.  People  vary 
consistently  in  terms  of  personality,  which  is  likely  to  have  a consistent  influence  on  group  interactions. 
Personality  influences  attitudes,  beliefs  and  behaviors  (Saucier  & Goldberg,  2003);  therefore,  an  individual’s 
personality  will  likely  influence  interactions  with  members  of  a community.  Despite  the  importance  of 
personality  in  learning  communities,  little  is  known  about  the  potential  influence  of  personalities  on  online 
discussion. 

Collaborative  learning  requires  communication  in  some  form.  Human  communication  involves  both 
cultural  and  social  processes.  The  cultural  process  reflects  learning  and  using  language  conventions  that  have 
shared  or  agreed-upon  interpretations.  Through  the  social  process,  communication  becomes  the  principal  way  in 
which  human  beings  experience  meaningful  interactions.  Through  such  interactions,  people  learn  to  play  roles, 
understand  social  norms,  recognize  and  apply  social  sanctions  and  evaluate  each  other’s  actions  according  to 
systems  of  shared  values  and  beliefs  (DeFleur  & BakRokeach,  1982;  Spears  & Lea,  1994).  Online 
communication  lacks  these  features  of  typical  human  communication  therefore  limiting  the  potential 
effectiveness  of  online  learning.  Knowing  that  personality  will  likely  influence  interactions  among  community 
members,  grouping  by  personality  may  compensate  for  these  missing  features.  Effective  grouping  strategies 
appear  to  be  the  essential  parts  of  the  online  learning  process  (Sherry,  1996,  2000;  Sherry,  Billig,  & Tavalin, 
n.d.).  However,  the  effect  of  grouping  by  personality  has  yet  to  be  investigated. 

Purpose  of  the  Study 

The  purpose  of  the  study  is  to  investigate  the  following  questions; 

1.  Does  personality  influence  group  interaction  in  terms  of  communication  type,  task  engagement,  and 
communication  duration? 

2.  Do  communication  pattern  differences  exist  among  groups  based  on  personality  profile? 

3.  Do  communication  patterns  change  over  time  due  to  personality  influences  on  group  interaction? 

4.  What  are  students’  feelings,  attitudes  and  opinions  about  their  online  discussion  experiences? 

Methodology 

Design 

A mixed  method  triangulation  design  was  used  to  address  the  questions  considered  in  this  study 
(McMillan,  2004). 
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Participants 

Seventy  undergraduate  education  majors  enrolled  in  three  sections  of  an  educational  psychology 
course  participated  in  the  study.  The  majority  (80%)  of  students  were  white  females  from  the  southeastern 
region  of  the  United  States.  Most  students  were  traditional  undergraduates  between  ages  18-22  enrolled  in  one 
of  their  first  education  courses.  Minority  and  non-traditional  students  were  also  enrolled  in  the  course  and 
participated  in  this  study. 

Instruments 

An  online  personality  inventory,  the  International  Personality  Item  Pool  (IPIP)  (Buchanan,  2001; 
Goldberg,  1999b),  was  used  to  classify  the  students’  personality  profiles.  Three  personality  traits  were  used  in 
this  study:  Extraversion,  a tendency  to  seek  and  engage  in  social  interactions;  Agreeableness,  reflecting  the 
quality  of  interaction  engagement;  and  Openness,  reflecting  an  interest  in  intellectual  and  imaginative 
experiences.  High  IPIP  scores  relative  to  the  norming  population  indicated  a greater  tendencies  toward  these 
personality  traits  (Saucier  and  Goldberg,  2001;  2002;  2003  ). 

Student  attitudes  about  their  online  learning  experiences  were  assessed  with  a paper  and  pencil  survey 
prepared  for  this  purpose  and  adminstered  at  the  end  of  the  semester. 

Procedure 

Students  were  assigned  to  one  of  the  four  personality  profile  groups  prior  to  the  start  of  the  online 
discussions.  Students  scoring  at  or  above  the  67th  percentile  of  the  sample  group  on  these  three  traits  were 
identified  as  “high,”  while  those  scoring  at  or  below  the  33rd  percentile  were  identified  as  “low.”  All  other 
students  were  considered  “neutral,”  that  is,  neither  predominantly  “high”  nor  “low”  in  the  relevant  traits. 
Therefore,  four  personality  profile  groups,  “high  profile,”  “low  profile,”  “high+low  profile,”  and  “neutral 
profile,”  were  formed,  including:  16  students  assigned  to  the  high  group,  9 assigned  to  the  low  group,  10 
assigned  to  the  high+low  group,  and  35  assigned  to  the  neutral  group,  for  a total  of  N=70. 

Students  participated  in  three  online  discussions  (approximately  one  per  month)  about  three  case 
studies  reflecting  course  concepts.  The  assignment  was  to  discuss  the  scenarios  using  relevant  course  concepts 
and  content,  as  well  as  open-ended  discussion  questions  provided  to  guide  the  conversations.  WebCT  (an  online 
course  management  system)  was  used  as  the  forum  for  online  discussions.  Discussions  had  specific  openning 
and  closing  dates.  Although  students  were  required  to  post  a minumum  of  three  messages  per  discussion,  some 
failed  to  do  so.  Therefore,  the  sample  sizes  varied  across  the  three  discussions  due  to  variations  in  the 
participation  rates. 

After  the  third  discussion,  students  completed  an  inclass  survey  regarding  their  feelings,  attitudes,  and 
opinions  about  using  online  discussion  and  collaborative  teamwork  to  complete  the  assignments. 

Data  and  Analyses 

The  content  of  each  message  was  analyzed  for  communication  type  (one-way  vs.  two-way),  task 
engagement  (fully  engaged;  somewhat  engaged;  disengaged),  and  communication  duration  (Rice  & Love, 

1987).  One-way  communication  (scored  1)  involved  messages  expressing  questions,  comments,  statements  or 
opinions,  but  neither  inviting,  encouraging,  nor  soliciting  reactions  from  group  members.  Two-way 
communication  (scored  2)  involved  inquiry  and/or  responding  messages  engaging  other  members  through 
questioning,  commentary,  statements  or  opinions  explicitly  responding  to  previous  messages  or  directly 
inviting,  encouraging  or  soliciting  replies.  Task  engagement  was  interpreted  as  the  degree  to  which  messages 
related  the  case  studies  and  course  concepts.  Being  “Fully  engaged”  (scored  3)  involved  content  that 
specifically  and  consistently  focused  on  instructional  or  assignment  issues.  “Somewhat  engaged”  (scored  2)  was 
used  to  code  content  that  clearly  but  inconsistently  reflected  instructional  or  assignment  issues,  and 
“Disengaged”  (scored  1)  reflected  content  that  was  either  marginally  related  or  unrelated  to  instructional  or 
assignment  issues. 

The  interrater  reliability  for  communication  was  .94  and  for  task  engagement  was  .91.  Communication 
duration  was  assessed  through  descriptive  analyses  of  word  count  per  message.  Student  attitude  was  measured 
using  a 6-point  Likert  scale.  Two  one-way  ANOVAs  were  conducted  to  examine  the  effects  of  personality  on 
communication  type  and  task  engagement.  Descriptive  statistics  were  used  to  examine  communication  patterns 
and  duration,  and  student  survey. 
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Results  and  Discussion 

Tables  1 and  2 present  the  results  related  to  communication  type.  Table  1 shows  descriptive  statistics 
for  all  groups  averaged  across  three  messages  for  each  discussion  and  across  all  three  discussions. 


Table  1:  Description  of  group  communication  type  across  three  discussions 


Group 

Discussion 

1 

Discussion 

2 

Discussion 

3 

Discussion 

overall 

High 

n 

13 

14 

14 

10 

Mean 

1.62 

1.57 

1.62 

1.63 

SD 

.36 

.30 

.34 

.20 

Low 

n 

8 

8 

8 

7 

Mean 

1.50 

1.08 

1.38 

1.30 

SD 

.47 

.15 

.33 

.21 

High+Low 

n 

8 

8 

10 

7 

Mean 

1.38 

1.71 

1.70 

1.59 

SD 

.28 

.21 

.189 

.106 

Neutral 

n 

25 

29 

32 

21 

Mean 

1.47 

1.52 

1.28 

1.41 

SD 

.236 

.374 

.282 

.234 

Total 

N 

54 

59 

64 

45 

Mean 

1.49 

1.50 

1.43 

1.47 

SD 

.316 

.358 

.33 

.23 

Table  2 shows  the  ANOVA  results  for  communication  type.  ANOVA  results  for  communication  type 
yielded  significant  differences  for  discussions  2,  3 (F(3,  55)=5.94,  pc. 001;  F(3,  60)=7.72,  pc. 001)  and  overall, 
F(3,  41)=  4.96,  pc. 01.  LSD  post  hoc  analyses  indicate  that  the  high-profile  group  consistently  engaged  in  more 
2- way  communication  than  both  the  low-  and  neutral-profile  groups.  Also,  the  high+low  group  outperformed 
both  the  low-  and  neutral-profile  groups,  suggesting  the  potential  personality  influence  of  the  high-profile 
members.  The  high-profile  group  consistently  engaged  in  two-way  communication  whereas  the  low-profile 
group  consistently  used  one-way  communication.  The  high+low  group  resembled  the  high-profile  group.  A 
gradual  change  in  communication  type  from  one-way  to  two-way  was  also  evident  for  this  group,  perhaps  due 
to  the  influence  of  high-profile  members.  The  neutral-profile  group  showed  a distinct  pattern  of  near  average 
performance. 


Table  2:  ANOVA  results  on  communication  type  across  three  discussions 


Discussion 

Source 

SS 

df 

MS 

F 

Sig. 

Discussion 

1 

Between 

Groups 

.32 

3 

.11 

1.09 

.362 

Within 

Groups 

4.95 

50 

.10 

Total 

5.28 

53 

Discussion 

2 

Between 

Groups 

1.82 

3 

.61 

5.94 

.001 

Within 

Groups 

5.6 

55 

.10 

Total 

7.42 

58 

Discussion 

3 

Between 

Groups 

1.96 

3 

.65 

7.72 

.000 

168 


Within 

Groups 

5.08 

60 

.09 

Total 

7.04 

63 

Discussion 

overall 

Between 

Groups 

.644 

3 

.22 

4.96 

.005 

Within 

Groups 

1.78 

41 

.04 

Total 

2.42 

44 

Tables  3 and  4 present  results  related  to  task  engagement.  Table  3 shows  the  descriptive  statistics  for 
all  four  groups  averaged  across  three  messages  for  each  discussion  and  across  all  three  discussions. 


Table  3:  Description  of  task  engagement  across  three  discussions 


Group 

Discussion 

1 

Discussion 

2 

Discussion 

3 

Discussion 

overall 

High 

n 

13 

14 

14 

10 

Mean 

2.41 

2.57 

2.43 

2.46 

SD 

.43 

.33 

.48 

.28 

Low 

n 

8 

8 

8 

7 

Mean 

2.54 

1.79 

2.63 

2.37 

SD 

.59 

.59 

.45 

.39 

High+Low 

n 

8 

8 

10 

7 

Mean 

2.71 

3.0 

2.83 

2.86 

SD 

.38 

.00 

.18 

.12 

Neutral 

n 

25 

29 

32 

21 

Mean 

2.53 

2.78 

2.59 

2.61 

SD 

.51 

.37 

.43 

.32 

Total 

N 

54 

59 

64 

45 

Mean 

2.53 

2.63 

2.60 

2.58 

SD 

.48 

.51 

.42 

.33 

Table  4 reports  the  ANOVA  results  for  task  engagement,  which  indicated  significant  differences  for 
discussion  2,  F(3,  55)=17.7,  pc. 001  and  overall,  F(3,  41)=  3.82,  p<. 05.  LSD  post  hoc  analyses  revealed 
significant  differences  at  the  .05  level  for  discussion  2 (high>low;  high+low>high  and  low;  neutral>low), 
discussion  3 (high+low>high)  and  overall  discussion  (high+low>high  and  low).  These  results  indicated  that  the 
high+low  group  appears  to  be  more  consistently  fully  engaged  than  the  other  three  groups. 

Table  4:  ANOVA  result  on  task  engagement  across  three  discussions 


Discussion 

Source 

SS 

df 

MS 

F 

Sig. 

Discussion 

1 

Between 

Groups 

.44 

3 

.15 

.62 

.606 

Within 

Groups 

11.9 

50 

.24 

Total 

4 

53 

Discussion 

2 

Between 

Groups 

7.42 

3 

2.48 

17.70 

.000 

Within 

Groups 

7.7 

55 

.14 

Total 

15.13 

58 

Discussion 

3 

Between 

Groups 

.96 

3 

.32 

1.85 

.148 

Within 

Groups 

10.41 

60 

.174 

169 


Total 

11.373 

63 

Discussion 

overall 

Between 

Groups 

1.04 

3 

.35 

3.82 

.017 

Within 

Groups 

3.71 

41 

.09 

Total 

4.76 

44 

The  comparison  of  communication  duration  averages  across  three  discussions  indicated  that  high-, 
high+low-,  and  neutral-profile  groups  exceed  the  message  length  of  the  low-profile  group,  but  the  high-profile 
group  consistently  surpassed  all  other  groups  until  discussion  3.  Table  5 shows  the  descriptive  statistics  for 
communication  durations  averaged  across  three  messages  for  each  discussion,  and  across  all  discussions.  In 
general,  high-profile  students  wrote  longer  messages.  In  overall  discussion,  the  high-profile  group  exceeded  the 
average  message  length  of  all  other  groups.  Additional  support  comes  from  the  high+low  group,  which  shows 
an  average  message  length  exceeded  the  low-  and  neutral-profile  groups. 

Table  5:  Mean  message  length  across  three  discussions 


Group 

Discussion 

1 

Discussion 

2 

Discussion 

3 

Discussion 

overall 

High 

n 

13 

14 

14 

10 

Mean 

136.72 

132.38 

100.76 

137.29 

SD 

73.38 

128.88 

42.06 

72.13 

Low 

n 

8 

8 

8 

7 

Mean 

100.63 

96.75 

82.79 

92.59 

SD 

39.04 

32.93 

22.92 

28.85 

High+Low 

n 

8 

8 

10 

7 

Mean 

114.92 

111.79 

105.3 

105.35 

SD 

29.23 

17.81 

34.41 

19.44 

Neutral 

n 

25 

29 

32 

21 

Mean 

109.45 

110 

108.6 

109.99 

SD 

56.08 

49.11 

47.2 

46.26 

Total 

N 

54 

59 

64 

45 

Mean 

115.52 

113.76 

103.15 

112.63 

SD 

55.84 

72.02 

41.94 

49.15 

Figure  1 compares  the  means  of  communication  duration  across  three  discussions  for  all  groups.  High-, 
high+low-,  and  neutral-profile  groups  all  far  exceed  the  message  length  of  the  low-profile  group,  but  the  high- 
profile  group  consistently  surpassed  all  other  groups  until  discussion  3. 
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Figure  1 : Patterns  of  communication  duration 


Group  mean  comparisons  of  communication  type  revealed  communication  patterns  that  distinguish 
these  personality  groups  (Figure  2).  The  high-profile  group  consistently  engaged  in  two-way  communication 
(78%  of  the  messages>1.5),  whereas,  the  low-profile  group  consistently  used  one-way  communication  (78%  of 
the  messages  <1.5).  The  high+low  group  resembled  the  high-profile  group  (67%  of  the  messages>1.5).  A 
gradual  change  in  communication  type  from  one-way  to  two-way  was  also  evident  for  this  group,  perhaps  due 
to  the  influence  of  high-profile  members.  The  neutral-profile  group  showed  a distinct  pattern  of  near  average 
performance  (67%  of  the  messages  were  between  1.4  and  1.6). 

Figure  2:  Patterns  of  communication  type  for  four  groups 


Table  6 reports  students’  feelings,  attitudes,  and  opinions  about  their  online  learning  experiences, 
grouped  into  five  categories.  The  data  indicated  that  regardless  of  personality  profile,  students  were  consistently 
positive  and  receptive  with  respect  to  their  online  discussion  experiences. 
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Table  6:  Results  of  students’  feelings,  attitudes,  and  opinions  about  online  learning  experience 


Category 

High 

(n=16) 

Low 

(n=9) 

High+Low 

(n=10) 

Neutral 

(n=32) 

Total 

(N=67) 

M(SD) 

M(SD) 

M(SD) 

M(SD) 

M(SD) 

Feeling  about  the  quality  of  online 

4.97 

4.94 

4.95 

4.75 

4.86 

discussion  tasks  & content 

(.78) 

(-92) 

(.88) 

(.93) 

(.89) 

Feelings  about  the  quality  of  online 

4.83 

4.89 

4.8 

4.4 

4.62 

collaborative  learning 

(.96) 

(.80) 

(.89) 

(1.13) 

(1.04) 

Attitudes  about  group  structure  & 

3.31 

3.78 

3.75 

3.63 

3.59 

membership 

(1.49) 

(1.26) 

(1.21) 

(1.35) 

(1.35) 

Opinions  about  assigning  a group  leader  in 

3.88 

3.56 

3.8 

3.5 

3.64 

the  discussion  form 

(1.63) 

(1.59) 

(1.62) 

(1.27) 

(1.43) 

Opinions  about  the  value  of  online 

4.78 

5.00 

4.65 

4.52 

4.66 

discussion  & professional  development 

(1.01) 

(.59) 

(.88) 

(1.13) 

(i.oi) 

Attitudes  toward  future  participation  in 

4.50 

4.67 

4.2 

4.16 

4.31 

forums  of  other  online  courses 

(1.59) 

(.71) 

(.92) 

(1.42) 

(1.32) 

Consistent  differences  in  communication  type  and  duration  were  identified  in  terms  of  personality 
profile.  Students  that  tend  to  be  more  socially  outgoing  and  engaging,  more  inclined  to  agreeableness  and 
seeking  of  intellectual  and/or  imaginative  experiences  seem  to  be  better  able  to  meet  the  goals  of  collaborative 
learning,  sharing  and  seeking  ideas,  comments,  questions  and  concerns,  as  well  as  engaging  in  longer  and  more 
sustained  conversations.  The  change  of  communication  type  from  one-way  to  two-way  for  the  high+low  group 
indicated  a possible  personality  influence  of  the  high-profile  peer  learners  in  the  same  group.  The  unexpected 
effects  of  personality  group  on  task  engagement  may  well  have  been  influenced  more  by  the  constraints  and 
structure  of  the  online  assignment,  which  was  somewhat  general  and  non-directive  with  respect  to  content  but 
highly  structured  in  terms  of  frequency  of  postings. 

Educational  Implications  and  Future  Research 

The  results  of  this  study  indicate  that  personality  affects  communication  type,  pattern  and  duration; 
productive  group  structures  result  in  sustained  discussion.  To  encourage  effective  online  communication, 
grouping  strategies  should  combine  personality  profiles  to  engage  rather  than  isolate  low-profile  students. 
Knowledge  of  personality  will  help  instructors  to  better  structure  the  collaborative  groups  and  to  develop  more 
dynamic  and  effective  online  learning  communities. 

Future  research  may  engage  in  a longitudinal  study  to  find  out  the  long-term  effects  of  grouping  by 
personality,  peer  influence,  and  stability  of  communication  patterns.  Future  studies  examining  the  extent  of 
conceptual  and  relational  structure  of  messages  will  illuminate  the  extent  to  which  personality  influences 
messages  content.  It  is  also  suggested  to  include  problem-  or  inquiry -based  tasks  in  the  learning  and  discussion 
in  future  studies.  Students  also  have  to  be  instructed  with  collaboration  strategies  in  order  to  work  effectively 
with  group  members. 
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The  Effects  of  Design  Strategies  for  Promoting  Students’  Self-regulated 
Learning  Skills  on  Students’  Self-Regulation  and  Achievements  in  Online 

Learning  Environments 
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University  of  Missouri-Columbia 

Abstract 

The  purpose  of  this  study  was  to  investigate  the  effects  of  design  strategies  for  promoting  students’ 
self-regulated  learning  skills  on  students’  self-regulation  and  achievements.  Seven  strategies  for  promoting 
students’  SRL  are  identified  through  the  literature  review  and  applied  into  the  experimental  group:  goal  setting, 
self-evaluation,  self-monitoring,  cognitive  strategies,  resource  management,  self-efficacy  and  volition.  Students 
were  assigned  into  the  control  and  experimental  group.  Independent  samples  T-test  and  semi -structured 
interview  were  conducted  to  analyze  the  effects  of  the  design  strategies.  Implications  to  promote  SRL  in  online 
learning  environment  were  discussed. 

Recently,  self-regulated  learning  (SRL)  has  emerged  as  an  important  issue  in  educational  circles 
(Boekaerts,  1999;  Schunk,  2000;  Schunk  & Zimmerman,  1998).  Self-regulated  learning  is  students’  active 
learning  processes  in  meta-cognition,  motivation,  and  behavior  (Zimmerman  & Martinez-Pons,  1986).  Self- 
regulated  learning  skills  are  critical  for  students  to  succeed  in  learning  not  only  in  traditional  learning 
environments,  but  also  in  web-based  learning  environments.  This  is  particularly  true  in  online  learning 
environments,  where  students  basically  learn  by  themselves  without  face-to-face  instruction  and  immediate  help 
from  teachers.  In  addition,  from  the  perspective  of  lifelong  learning,  the  needs  for  E-learning  have  been 
increasing.  This  relatively  unfamiliar  learning  environment  can  be  challenging  to  students.  Therefore, 
promoting  students ' SRL  skills  is  something  that  instructional  designers  should  consider  when  they  design 
online  learning  courses. 

Zimmerman,  Bonner,  and  Kovach  (1996)  argued  that  students'  self-regulation  can  be  taught  and 
improved  through  the  students’  own  efforts.  However,  promoting  students’  self-regulation  is  not  an  easy  task 
because  it  requires  them  to  spend  a lot  of  time  and  energy.  In  addition,  promoting  self-regulation  is  only 
possible  when  students  experience  the  benefits  of  self-regulation  (Zimmerman,  Bonner  & Kovach,  1996). 

Many  researchers  argued  that  the  effective  way  to  improve  students’  SRL  skill  is  to  embed  SRL 
strategies  into  the  context.  This  is  because  students  do  not  apply  the  learned  SRL  skills  into  their  learning 
context  after  they  learned  self-regulated  learning  skills.  Also,  it  is  important  to  have  students  experience 
(Zimmerman,  Bonner  & Kovach,  1996)  and  use  the  designed  SRL  skills  into  their  learning.  It  is  true  that  many 
students  even  don  ’t  click  a designed  content  or  button  and  ignore  many  important  learning  events  designed  for 
them  (Lim,  2002).  Many  researchers  (Ley  & Young,  2001;  Zimmerman,  Bonner  & Kovach,  1996)  suggested  the 
following  four  design  principles  to  promote  students  ’ self-regulated  learning  skills:  (1)  The  SRL  activities  need 
to  be  explicitly  delivered  to  students.  (2)  Students  should  have  opportunities  to  utilize  learned  SRL  strategies  in 
real  learning  situations.  ( 3 ) Intervention  to  promote  students ' SRL  skills  should  be  mandatory  or  strongly 
structured.  (4)  Having  students  successfully  experience  SRL  skills  is  needed  for  regular  application  of  SRL 
skills  in  their  actual  learning. 

What  self- regulated  learning  skills  are  critical? 

Self -regulated  learning  strategies  consist  of  cognitive  and  meta-cognitive  activities,  resource 
management  activities,  and  affective  activities  (Zimmerman  and  Martinez,  1986;  Pintrich,  1999).  Corno  and 
Mandinach  (1983)  viewed  self -regulated  learning  as  a deliberate  planning  and  monitoring  process  and 
emphasized  the  importance  of  cognitive  and  meta-cognitive  activities  for  self -regulated  learning.  Cognitive 
activities  refer  to  rehearsal,  elaboration,  and  organization  (Hofer,  Yu,  and  Printrich,  1998;  Yang,  2000). 
According  to  Printrich  (1999),  rehearsal  strategies  are  recitation  of  items  to  be  learned,  saying  the  word  aloud 
when  students  read,  and  highlighting  or  underlining  the  text,  elaboration  strategies  are  paraphrasing  or 
summarizing  the  material,  and  organizational  strategies  are  selecting  the  main  ideas  and  outlining  the  text. 
Cognitive  activities  vary  depending  on  the  learning  domain. 

With  cognitive  activities,  meta-cognitive  activities  are  critical  for  self-regulated  learning.  If  cognitive 
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activities  are  specific  strategies  to  accomplish  goals,  meta-cognitive  strategies  are  monitor  and  reflection  to 
accomplish  goals.  Meta-cognitive  activities  are  goal  setting,  self- monitoring,  and  self-evaluation.  Goal  setting 
refers  to  deciding  on  specific  learning  outcomes  (Zimmerman,  1999).  Schunk  (2000)  mentioned  two  roles  of 
goal  setting:  a motivator  to  exert  persistent  effort  over  time  and  the  criteria  to  monitor  learning  progress.  A self- 
monitoring activity  involves  comparing  goals  and  current  accomplishments  with  the  use  of  cognitive  activities. 
There  are  a lot  of  self -monitoring  methods  depending  on  learning  context,  e.g.,  narrating  behavior  in  the 
context,  recording  frequency  counts,  measuring  duration,  time  -sampling  (dividing  observation  periods  into 
smaller  time  periods),  and  tracing  times  of  behavior  occurring  (Mace,  Belfiore,  and  Hutchinson,  2001).  For  an 
effective  self- monitoring,  it  should  regularly,  proximately,  (Schunk,  2000)  and  accurately  (Mace,  Belfiore,  and 
Hutchinson,  2001)  occurred.  Self-evaluation  is  a learners’  judgment  on  their  performance.  Self- monitoring 
plays  a great  role  in  self-evaluation.  Based  on  the  results  of  comparing  performance  to  standards  or  goals,  self- 
regulated  learners  decide  to  whether  they  will  change  cognitive  strategies,  keep  going  the  efforts,  or  give  more 
efforts.  Self-evaluation  and  self-monitoring  occur  almost  at  the  same  time. 

Resource  management  activities  are  time  and  effort  management,  seeking  help  from  others,  seeking 
information  and  structuring  environment  for  learning  (Pintrich,  1999).  Resource  management  activities  can 
occur  differently  depending  on  what  prior  knowledge  about  subjects  students  have  and  what  resources  they  can 
use  in  their  context.  The  activities  for  resource  management  are  not  directly  related  to  cognitive  and  meta- 
cognitive  activities  (Pintrich,  1999)  but  they  are  important  for  academic  success  (Hofer,  Yu,  and  Pintrich, 

1998).  Zimmerman  and  Martinez’s  research  (1986)  also  indicated  that  high  self-regulated  learners  did  resource 
management  activities  more  frequently  than  low  self-regulated  learners  did. 

In  addition  to  cognitive,  meta-cognitive  and  resource  management  activities,  students’  affective 
activities  play  a significant  role  for  their  self-regulated  learning  (Pintrich,  1999;  Schunk  and  Zimmerman,  1998; 
Shin,  1998).  Self-efficacy  is  students’  confidence  about  their  ability  to  perform  a task.  Scott  (1996)  found  that 
high  self-efficacy  students  tend  to  be  confident  and  motivate  themselves  to  acquire  learning  while  low  self- 
efficacy  students  tend  to  less  motivate  themselves  to  learn  and  think  that  acquiring  goals  are  difficulty.  Parajes 
(2002)  found  that  high  self-efficacy  students  tend  to  exert  more  effort  than  low  self-efficacy  students  do  when 
they  meet  obstacles  in  learning.  With  self-efficacy,  volition  is  also  important  for  self -regulated  learning 
(Garcia,  McCann,  Turner,  and  Roska,  1998;  Kehr,  Bles,  and  Rosenstiel,  1999;  Kuhl,  2000).  Volition  is  students’ 
will  power  to  accomplish  certain  goals.  Garcia,  McCann,  Turner,  and  Roska  found  that  volition  is  strongly 
related  to  students’  use  of  cognitive  and  resource  management  activities.  They  argued  that  volition  leads 
students  to  goal-directed  learning  and  teaching  volitional  skills  to  students  will  be  helpful  for  them  to  self- 
regulated  learners. 

How  self- regulated  learning  strategies  are  designed  in  online  learning  environment? 

Seven  self-regulated  learning  strategies  are  embedded  in  the  context  for  learning  the  Test  of  Written 
English  (TWE).  Learners  are  required  to  practice  every  designed  SRL  skill  in  each  chapter.  When  practicing 
cognitive,  meta-cognitive,  resource  management  and  affective  activities,  students  are  asked  to  submit  the  results 
of  the  each  activity  to  the  instructor. 

Meta-cognitive  activities:  regarding  goal  setting  students  were  asked  to  hierarchically  set  the  goals  for 
the  course  at  the  beginning  of  the  class.  Students  set  the  goals  by  answering  the  questions:  how  this  course 
contributes  to  getting  a job  in  the  future,  what  goals  you  have  after  one  year  with  regards  to  this  course  and  what 
goals  you  have  after  one  month  with  regards  to  this  course.  In  addition,  they  are  asked  to  write  down  what 
distracts  their  learning  and  devise  a plan  to  overcome  the  problems.  With  the  regard  of  self- monitoring,  students 
were  asked  to  self-monitor  by  checking  learning  processes  box.  The  questions  in  the  box  to  check  are  asking 
about  their  goal  achievement  and  using  cognitive  and  resource  management  strategies.  Last,  in  order  to  promote 
students’  self-evaluation,  writing  journal  was  required  of  the  students. 

Cognitive  activities:  rehearsal,  elaboration,  organization  strategies  are  suggested  as  learning  clues  on 
the  screen  with  the  feedback  format  whenever  it  is  necessary.  Students  were  asked  to  practice  suggested 
cognitive  strategies  and  submit  the  results  to  the  instructor. 

Resource  management  activities:  Before  starting  learning,  structuring  learning  environment  questions 
are  given  to  students  with  the  checkbox  format.  The  questions  are  about  whether  they  organized  a learning 
environment  for  the  learning.  They  were  also  asked  to  submit  their  time  schedule  for  this  course,  e.g.,  how  they 
schedule  this  course  in  their  daily  lives  and  how  much  time  they  will  spend  on  the  course.  Last,  for  effective 
help,  a help  desk  was  operated  through  the  discussion  board. 

Affective  activities:  Feedback  was  given  on  every  student’s  assignment.  When  giving  feedback. 
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attribution  feedback  is  given  with  compliment,  e.g.,  “Your  writing  is  good.  If  you  keep  this  pace,  your  writing 
will  be  greatly  improved”.  In  addition  to  attribution  feedback,  volition  encouragement  was  given  with  the 
learning  strategy  clues  on  the  screen.  With  structuring  learning  environment  checkbox,  questions  asking 
volition  were  given  to  remind  them  of  the  importance  of  their  volition  in  learning.  The  SRL  design  strategies 
are  summarized  in  Table  1. 

Table  1.  Design  strategies  to  promote  self-regulated  learning 
SRL  strategies  Design  strategies 

Meta-cognitive  activities  Goal  setting:  students  set  goals  after  reading  each  chapter  overview 

Self- monitoring:  students  check  their  learning  process  at  the  end  of  each 
chapter 

Self-evaluation:  students  write  a journal  about  their  learning 
Cognitive  activities  Rehearsal,  elaboration,  organization:  Necessary  rehearsal,  elaboration, 

organization  are  suggested  to  students  as  a feedback  form  depending  on 
content. 

Resource  management  Time  management:  students  are  asked  to  plan  their  time  for  learning 

activities  Help  seeking:  Online  help  seeking  corner  is  constructed  and  encourage 

students  to  use  it  any  problem  related  to  learning 

Structuring  learning  environment:  Feedback  is  given  to  students  before  starting 
each  chapter 

Affective  activities  Self-efficacy:  Progression  and  attribution  feedback  are  given  to  encourage 

them  to  learn  and  keep  going  their  learning 

Volition:  students  check  their  volition  before  they  learn  each  chapter  and 
feedback  encouraging  volition  is  given 

Course  Development 

The  course  consisted  of  12  lessons,  and  the  experiment  was  conducted  for  a month.  Two  online 
learning  sites  for  the  control  group  and  the  experimental  group  were  respectively  developed  to  verify  the 
effectiveness  of  design  strategies  for  promoting  self -regulation  with  the  use  of  the  book,  “To  Be  A Master  In 
TWE”  (Min,  2002).  The  online  TWE  (Test  of  Written  English)  program  used  in  the  control  group  was 
developed  according  to  the  Gagne’s  nine  events.  Another  website  for  the  experimental  group  was  developed 
according  to  the  devised  SRL  strategies.  Both  groups’  students  commonly  should  submit  their  assignments 
three  times  per  week.  In  addition  to  turning  in  the  assignments,  the  students  in  the  experimental  group  should 
obligatorily  practice  Self-regulated  Learning  activities  and  post  the  results  on  the  online  bulletin  boards  for  each 
class.  SRL  strategies  are  visualized  in  figure  1. 

In  the  figure,  the  upper  menu  involves  a learning  preparation,  learning  overview,  learning  goals, 
learning  content,  learning  evaluation,  and  learning  arrangement.  The  bottom  of  the  program  menu  involves  a 
syllabus,  room  for  submitting  assignments,  notice,  and  learning  aids  consisting  of  button  explanations,  asking 
questions,  total  dictionary,  summary  of  important  terminologies,  and  online  English  learning  sites.  In  addition, 
five  buttons  for  SRL,  used  for  setting  course  goals,  planning  learning  resources,  establishing  learning  goals, 
following  learning  strategies,  and  writing  a reflective  diary  are  incorporated  on  the  right  side  of  the  screen. 

Data  gathering 

Thirty  students  in  a Korean  university  volunteered  for  this  research  for  a month.  Most  of  the  students 
were  freshmen.  The  students  are  randomly  assigned  into  the  experimental  group  or  the  control  group  using 
random  numbers . This  is  a pre  and  post  test.  Students  ’ self-regulated  learning  skills  and  essay  levels  were 
measured  before  and  after  the  treatment. 

The  Self- Regulated  Learning  Strategies  Questionnaire  developed  by  Yang  (2000)  was  used  to  measure 
students  ’ SRL  level.  The  SRL  questionnaire  consists  of  cognitive,  meta-cognitive,  motivational  and  behavior 
strategies.  The  number  of  item  is  84.  It  uses  a self-reported  five-Likert  scale. 

An  essay  topic  randomly  chosen  from  the  ETS  TWE  topics  is  used  to  measure  students  ’ prior  knowledge  and 
achievements.  These  were  measured  by  providing  students  a topic  and  letting  them  write  an  essay  about  the 
topic.  The  criteria  used  in  ETS  were  also  used  in  rating  students  ’ prior  knowledge  and  achievements.  ETS  uses 
0-6  scale  points  to  evaluate  students’  essay  where  6 is  the  best  score. 
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Research  Questions 

Students  studying  in  learning  environments,  which  are  designed  to  forcefully  encourage  the  practice  of 
SRL  skills,  will  show  a higher  self -regulation  than  others  studying  in  normal  learning  environments,  which 
don’t  support  SRL  activities. 

Students  studying  in  learning  environments,  which  are  designed  to  forcefully  encourage  the  practice  of 
SRL  skills,  will  show  a higher  achievement  than  others  studying  in  normal  learning  environments,  which  don’t 
support  SRL  activities. 


Results 

Independent  samples  T test  for  group  comparison  and  semi -structured  interviews  were  used  to  analyze 
the  data.  Pre-test  results  showed  that  there  is  no  significant  difference  in  SRL  between  two  groups.  Also,  there 
was  no  significant  difference  in  TWE  level  between  two  groups.  The  mean  scores  of  each  group  was  the  same 
with  each  other,  M = 1.067. 

SRL  Post-test  indicated  that  there  was  no  significant  difference  between  groups.  Also,  there  was  no 
significant  different  in  SRL  strategies  between  groups.  Experimental  group’s  sum  of  SRL  scores  (276.85)  were 
slightly  higher  than  those  of  control  group  (274.07).  However,  it  was  not  significantly  different.  This  means  that 
the  treatment  having  students  practice  SRL  skill  was  not  effective. 


Table  2.  SRL  level  comparison  between  experimental  and  control  group. 


Control  group 

Experimental  group 

M 

SD 

M 

SD 

Cognitive 

58.73 

6.11 

59.47 

6.19 

Meta-cognitive 

36.20 

4.92 

35.27 

4.83 

Motivational 

87.47 

14.41 

86.27 

10.43 

Behavioral 

91.67 

12.93 

95.87 

7.97 

SRL 

274.07 

30.44 

276.87 

20.62 

Regarding  the  students’  TWE  levels,  there  was  slight  difference  between  the  two  groups.  The  mean  score  of  the 
experimental  group  (M  = 3.07)  was  slightly  higher  than  mean  of  control  group  (M  = 2.97).  However,  the 
difference  was  not  significant.  The  one  important  thing  is  that  TWE  scores  were  significantly  improved  during 
the  one  month  in  both  groups,  t (29)  = -20.761,  p = .00  (two-tailed).  This  means  that  the  online  course  was 
effective  to  improve  students’  performance. 

In  order  to  identify  the  reasons  why  the  treatment  was  not  effective,  semi -structured  interviews  with 
the  experimental  group  students  were  conducted.  The  interview  data  revealed  that  students  didn’t  know  how  to 
effectively  practice  the  intended  SRL  skills  and  they  didn’t  know  the  necessity  of  practicing  SRL  skills.  Many 
students  in  the  experimental  group  felt  that  practicing  SRL  skills  was  another  assignment  which  made  them 
annoying.  Many  students  reported  that  the  designed  SRL  practices  were  demanding.  In  the  experimental  group, 
students  were  required  to  submit  every  result  of  SRL  practices  three  times  a week.  This  fact  made  them  less 
motivated  in  learning  TWE.  Last,  they  felt  that  individualized  SRL  practice  were  necessary.  Some  of  them  were 
already  good  at  cognitive  activities  or  time  planning.  They  did  not  want  to  follow  practices,  which  were 
different  from  their  own  ways.  The  interview  data  gave  clues  why  the  treatment  was  not  effective  to  promote 
their  SRL  skills. 


Discussion 

The  purpose  of  this  research  was  to  investigate  the  design  strategies  for  promoting  SRL  skills  on 
students  ’ SRL  skills  and  performance.  The  research  results  imply  three  things  to  consider  when  designing  SRL 
practice  and  training  SRL  skills  in  online  learning  environments. 

First,  college  level  students’  self-regulated  learning  skills  are  not  something  to  be  improved  in  short 
time  periods  just  by  forcefully  having  them  practice  activities.  Interview  results  showed  thatstudents  felt  a lot 
of  burden  because  of  the  mandatory  participation  in  every  designed  self-regulated  learning  activity.  This  led 
some  students  ’ motivation  going  down  and  being  hesitant  to  use  self-regulated  learning  activities. 

Second,  exposing  students  to  practice  self-regulated  learning  skills  is  not  enough  to  promote  their  self- 
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regulated  learning.  They  need  continuous  interactions  with  peers  or  with  instructors  about  their  progress. 
Interview  results  showed  that  many  students  was  not  able  to  fully  understand  the  purpose  of  self -regulated 
learning  and  why  they  were  doing  the  activities.  The  interactions  with  others  will  remind  them  to  think 
continuously  about  their  activities  and  progresses.  This  will  lead  them  to  self-regulated  leaner  and  to  apply  the 
acquired  skills  to  other  contexts. 

Third,  autonomy  and  responsibility  should  be  given  to  students  to  self-regulate  their  own  learning 
while  they  practice  designed  practices.  The  online  program  was  intended  to  give  as  many  opportunities  for 
students  to  practice  self-regulated  learning  skills  and  feel  the  benefits  of  them.  That’s  why  it  demanded 
students  ’ mandatory  participation  in  the  SRL.  However,  it  did  not  consider  how  students  ’ self-regulated 
learning  skills  are  different.  For  example,  some  students  are  good  at  resource  managements  while  they  are  not 
good  at  cognitive  activities.  Some  students  are  good  at  meta-cognitive  activities  while  they  are  not  good  at 
resource  managements.  By  allowing  some  extend  autonomy  and  responsibilities  they  will  focus  on  their 
weakness  of  self-regulated  learning  skills. 


Figure  1.  Self -regulated  learning  design  strategies  in  the  online  learning  environment 
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Introduction 

The  research  team  identified  and  tested  six  criteria  that  operationalize  cultural  responsiveness  as  a 
factor  for  evaluating  school  district  Websites:  cultural  utility,  representation  of  diverse  cultures  of  target 
populations  in  graphics,  percentage  of  website  topics  translated  into  Spanish,  the  content  of  topics  translated, 
the  level  on  which  translation  takes  place,  and  the  level  on  which  website  navigation  elements  are  translated 
into  Spanish.  Fifty  school  district  Websites  were  evaluated.  Findings  indicate  lack  of  cultural  responsiveness  on 
websites  and  that  cultural  responsiveness  should  be  added  to  website  evaluation  instruments. 

Problem 

School  districts  serve  not  just  their  students,  but  also  the  communities  where  students  live.  In  addition  to  being 
educational  institutions,  they  are  cultural,  political,  and  economic  institutions.  Therefore,  school  districts  need  to  address 
cultural  responsiveness  on  their  websites  (Huang  & Tilley,  2001;  Badre  & Barber,  1998).  On  websites,  “metaphors, 
mental  models,  navigation,  interaction,  or  appearance  confuse,  or  even  offend  and  alienate,  a user”  (Marsus  & Gould, 
2000).  The  need  for  Texan  school  district  Websites  to  address  language  and  worldview  differences  for  the  Latino 
community  provides  a basis  for  asking,  “How  culturally  responsive  are  public  school  district  Websites  to  the  needs  of  the 
Spanish  speaking  community?” 

Ninety-five  percent  of  the  families  of  the  630,000  students  that  have  been  placed  in  Limited  English 
Proficient  programs  in  Texas  indicated  that  they  speak  Spanish  in  the  home.  In  addition,  many  more  families 
have  indicated  that  Spanish  is  spoken  in  the  home  but  have  not  placed  their  children  in  the  Limited  English 
Proficiency  program  (Seidner,  Director  of  the  Bilingual  Education  for  the  Texas  Education  Agency,  personal 
communication,  July  27,  2003).  Although  federal  law  assures  equitable  access  to  educational  resources,  Spanish 
speakers  in  Texas  have  limited  access  to  general  information  that  is  provided  on  school  district  Websites. 

Website  evaluation  typically  focuses  on  the  dimensions  of  design,  credibility,  usability,  and  accessibility 
(Alexander  & Tate,  1999;  Lynch  & Horton,  2001;  Nielsen,  1993).  Design  relates  to  the  overall  aesthetics  of  a website. 
Credibility  involves  the  quality  of  the  information  on  a website  (Fogg,  et  al,  2001).  Usability  refers  to  the  ease  with 
which  users  find  information  on  a website  and  their  level  of  satisfaction  with  the  experience.  Accessibility  is  the  use  of 
the  Internet  by  people  with  disabilities  through  assistive  technologies  such  as  web-readers  for  the  blind  or  pointing  sticks 
for  the  physically  handicapped. 

Cultural  responsiveness  in  website  design  impacts  each  of  these  dimensions  and  is  particularly  important  for 
equitable  access  to  educational  environments.  The  emergence  of  Latinos  as  a force  in  online  shopping  and  political  polls 
has  triggered  acknowledgment  by  the  business  and  political  worlds  of  the  importance  of  tailoring  websites  to  reach 
Spanish  speaking  populations  (Swartz,  2003).  However,  our  literature  review  reveals  that  educational  institutions  are  not 
yet  addressing  cultural  responsiveness  on  Websites  and  that  cultural  responsiveness  is  not  included  as  a Website 
evaluation  criterion  on  existing  instruments. 


Objectives 

Three  intents  of  this  exploratory  study  were  to  (1)  develop  criteria  for  evaluating  the  effectiveness  of  school 
district  websites  at  meeting  the  needs  of  the  Spanish  speaking  community,  (2)  field  test  these  criteria  using  50  district 
websites  in  Texas,  and  (3)  specify  directions  for  future  efforts  aimed  at  website  evaluation  to  address  cultural 
responsiveness. 


Participants 

Six  criteria  for  evaluating  websites’  cultural  responsiveness  were  tested  on  websites  for  the  50  largest  school 
districts  in  Texas  (a  purposive  sample)  using  both  descriptive  statistics  and  naturalistic  inquiry  methods  (Lincoln  & 
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Guba,  1985).  These  included  districts  in  Houston,  Dallas,  Fort  Worth,  San  Antonio,  Austin,  El  Paso,  Brownsville,  and 
Laredo.  They  represent  50%  of  the  total  student  population  in  the  state  of  Texas  and  75%  of  the  limited-English- 
proficient  student  population.  The  six  criteria  tested  were:  Percentage  of  topics  translated  to  Spanish,  the  content  of 
topics  translated  to  Spanish,  level  on  which  translation  first  takes  place,  level  on  which  navigation  devices  (menus  and 
navigation  bars  that  enable  movement  through  several  topics  on  the  Web  site)  are  translated  to  Spanish,  cultural  utility, 
and  inclusiveness  of  graphics. 


Methods 

Criteria  for  evaluating  websites’  cultural  responsiveness  were  tested  on  Texan  school  districts  using 
both  descriptive  statistics  and  naturalistic  inquiry  methods  (Lincoln  & Guba,  1985).  The  field  test  evaluated  the 
websites  of  the  50  largest  school  districts  in  Texas.  These  included  districts  in  Houston,  Dallas,  Fort  Worth,  San 
Antonio,  Austin,  El  Paso,  Brownsville,  and  Laredo  which  represent  50%  of  the  total  student  population  in  the 
state  of  Texas  and  75%  of  the  limited- English-proficient  student  population. 

Evaluation  Matrices  were  designed  to  describe  each  Website  in  terms  of  six  cultural  responsiveness  criteria: 

♦ Cultural  utility  was  estimated  by  establishing  topics  of  interest  to  parents  in  the  Latino  community 
and  determining  which  of  those  was  addressed  and  how  many  of  the  topics  were  translated. 
Community  interests  were  identified  through  interviews  with  individuals  and  focus  groups  with 
Latino  parents  and  students.  These  included  a representative  from  the  Mexican  American  Legal 
Defense  and  Educational  Fund,  a representative  from  the  Mexican  American  Latino  Research 
Center  at  a research  university,  and  six  focus  groups  of  parents  and  teachers  from  different  school 
districts  in  Central  Texas  and  from  the  Rio  Grande  Valley.  Each  reacted  to  a variety  of  websites 
revealing  their  interests  and  reactions,  both  positive  and  negative.  The  interview  process  was 
conducted  by  two  of  the  researchers  and  contents  were  recorded  and  transcribed  for  evaluation  by 
all  researchers  who  identified  emergent  themes  by  color  coding  text  and  marking  in  margins.  We 
achieved  consensus  regarding  topics  of  interest  to  Spanish  speaking  users  and  conclusions  that 
could  be  drawn  regarding  culturally  responsive  design  issues. 

♦ Percentage  of  topics  translated  to  Spanish  : The  researchers  counted  the  topics  listed  on  levels  one 
and  two  of  each  site,  counted  the  topics  translated  on  those  two  levels,  and  calculated  the 
percentage  translated.  If  a topic  that  appeared  on  the  first  level  appeared  again  on  the  second  level, 
it  was  not  counted  twice.  If  translation  was  indicated  on  the  second  level,  then  the  researchers 
pursued  those  topics  to  their  full  extent.  Therefore,  most  topics  that  were  translated  on  subsequent 
levels  were  identified. 

♦ The  content  of  topics  translated  to  Spanish:  These  will  be  identified  during  the  exploratory  process 
described  for  criteria  one  and  will  include  translated  topics  found  on  any  level  of  the  site. 

♦ Level  on  which  translation  first  takes  place. 

♦ Level  on  which  navigation  devices  are  translated  to  Spanish  (menus  and  navigation  bars  that 
enable  movement  through  several  topics  on  the  Web  site). 

♦ Inclusiveness  of  graphics  was  determined  by  six  evaluators.  The  evaluators  all  had  college  degrees 
in  education,  business,  computer  science,  or  biology.  They  ranged  in  age  from  21  to  50  and 
included  4 females  and  2 males.  All  six  evaluators  were  white.  They  listed  graphics  on  each 
district’s  home  page  and  labeled  each  listed  graphic  as  cross-cultural,  neutral,  or  specific.  If 
evaluators  labeled  a graphic  as  “specific,”  they  specified  the  culture  represented  by  that  graphic. 
They  provided  a rationale  for  the  decision  making  regarding  culture  represented  by  graphics.  After 
compiling  evaluators’  categorizations  and  rationales  in  a summary  matrix,  we  paid  closest 
attention  to  the  evaluators’  rationales.  Throughout  the  data  analyses  we  inserted  our  own  judgment 
and  understanding  to  summarize  and  draw  conclusions  keeping  in  mind  that  the  ambiguous  nature 
of  the  data  made  it  subject  to  our  interpretation.  Therefore,  we  do  not  claim  that  our  interpretation 
is  the  sole  interpretation  of  these  data. 


Results 

Preliminary  results  of  the  exploratory  study  follow.  In  testing  the  six  criteria  on  Texas  ISD  Web  sites, 
we  found  that  they  informed  us  regarding  dimensions  of  cultural  responsiveness. 

Focus  groups  and  interviews  of  the  Latino  community  members  revealed  that  cultural  utility,  or  topics 
of  interest,  included  information  about  registration,  bussing,  curriculum  and  programs  of  study,  quality  of 
program,  class  sizes,  immunization  information,  how  to  contact  the  Board  of  Trustees,  how  to  make  course 
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changes,  extra  curricular  activities,  wages  of  teachers,  and  credentials  of  teachers. 

For  31  ISD  Websites,  the  percentage  of  topics  translated  to  Spanish  was  .02%  of  the  topics  posted  on 
school  district  Websites  . They  ranged  from  0 translation  to  .15%  of  topics  translated.  However  one  site  was  an 
outlier  with  23%  of  the  610  topics  translated.  This  site  was  not  included  in  the  averaging  process. 

The  content  of  topics  translated  to  Spanish  Translated  content  was  idiosyncratic  in  that  it  varied  widely 
from  district  to  district  including  topics  ranging  from  bacterial  meningitis  warnings  to  parental  permission 
forms.  Generally  content  fell  under  the  following  categories:  Calendar,  newsletters,  menu,  PTA  information, 
school  handbook,  health  information,  bond  information,  enrichment  information,  and  curriculum  Curricular 
information  included  information  about  migrant  education,  bilingual  programs , pre  -kindergarten,  kindergarten, 
recommended  high  school  programs,  magnet  schools,  grading  periods,  exam  schedules,  and  gifted  and  talented 
programs.  Only  two  of  the  31  sites  had  translated  information  about  gifted  and  talented  programs. 

Again  for  just  31  sites,  the  level  on  which  translation  first  took  place  were  identified.  Translations  were 
typically  found  on  the  third  or  fourth  level  of  the  Websites  making  them  inaccessible  to  users  who  only  spoke 
Spanish.  One  site  had  no  translation;  6 sites  began  their  translations  on  level  3,  and  14  sites  began  their 
translations  on  level  2.  Ten  sites  had  an  “En  EspanoF'  on  the  first  level  of  the  site. 

For  those  same  31  sites,  navigation  devices  were  translated  to  Spanish  starting  with  different  levels. 

One  site  had  no  translation  for  navigation;  7 sites  began  their  translations  on  level  3,  and  13  sites  began  their 
translations  on  level  2.  Ten  sites  had  the  “En  EspanoF'  link  on  the  first  level  of  the  site. 

Website  developers  appear  to  be  somewhat  responsive  in  their  selection  of  images  that  revealed  the 
cultural  diversity  of  the  district.  On  the  50  Websites,  one  hundred  and  eleven  graphics  were  polycentric,  707 
were  culturally  neutral  in  their  imagery,  and  5 1 were  ethnically  specific  or  ethnocentric. 

Data  analysis  reveals  that  the  criteria  developed  for  website  evaluation  facilitates  identification  of  cultural 
responsiveness  or  unresponsiveness  on  websites. 

Conclusions 

When  evaluating  web  sites,  cultural  responsiveness  should  be  considered  as  an  aspect  of  each  of  the  four 
conventional  criteria:  design,  credibility,  usability,  and  accessibility.  Continued  research  needs  to  aim  at  further 
identifying  clear  operational  elements  of  culturally  responsive  website  design. 

A next  step  is  to  quantify  our  developed  and  tested  criteria.  Once  our  criteria  have  been  quantified,  a cultural 
responsiveness  score  and  portrait  of  any  district  website  will  be  able  to  be  drawn.  Based  upon  each  score  and  portrait, 
specific  recommendations  for  improving  cultural  responsiveness  in  website  design  can  be  specified  for  any  given  Web 
site. 
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Online  learning  has  been  hailed  as  a huge  step  forward  for  education.  Academia  is  increasingly 
offering  Internet-based  distance  courses  because  they  allow  “anywhere,  anytime”  learning  and  access  to  a wider 
pool  of  students.  Firms  and  organizations  like  it  for  training  employees  because  it  permits  consistency  and 
repeatability  in  training  delivery.  K-12  educators  endorse  it  because  it  teaches  children  important  technology 
skills  that  they  will  need  in  their  advanced  education  process  and  in  their  careers. 

Online  learning  has  been  imp  licitly  considered  the  lingua  franca  of  education  in  terms  of  its 
functioning  and  graphic  content  (Gunawardena  & Mclsaac,  2004,  p.363).  For  the  most  part,  instructional  design 
of  online  learning  has  largely  ignored  culture  in  the  creation  of  online  learning  environments  (Thomas,  2002). 

By  not  taking  learners’  culture  into  consideration,  it  is  possible  that  online  learning  environments  have 
been  designed  in  such  a way  as  to  negatively  impact  learners’  motivation  and  persistence  levels.  For  example, 
Merryfield  (2001)  found  unexpected  challenges  in  transferring  a course  on  diversity  in  education  to  the  Internet. 
She  observed  that  many  of  the  behavioral  aspects  that  differentiated  students  were  broken  along  cultural  lines, 
and  noted  that  this  may  have  implications  for  equity  of  access. 

In  the  European  Union,  where  issues  of  cultural  diversity  loom  even  larger  than  they  do  in  American 
classrooms,  there  have  already  been  some  efforts  to  find  an  acceptable  middle  ground  in  educational  software 
and  to  make  it  more  portable  across  political  and  ethnic  borders  (Collis  & Remmers,  1997).  Suggestions  offered 
by  the  Commission  of  the  European  Communities  are  pragmatic  and  aim  at  making  educational  software 
equally  usable  by  all,  but  they  do  not  address  issues  of  learner  motivation  except  by  inference. 

The  fact  that  online  learning  and  distance  education  can  impose  its  authors’  goals,  perspectives,  and 
standards  on  a receiving  culture  (Gunawardena  & Mclsaac,  2004,  p.  388)  is  very  rarely  cited  in  the  literature. 
Understanding  whether  and  how  learners’  cultural  characteristics  may  interact  with  a major  delivery  method 
such  as  online  learning  makes  possible  the  exploration  and  creation  of  alternative  means  of  supporting  learners 
in  the  construction  of  knowledge.  Not  doing  so  may  well  condemn  groups  other  than  the  course  implementer’s 
or  author’s  to  decreased  motivation  and  thus  higher  attrition,  reducing  some  learners’  chances  of  benefiting 
equally  from  a digital  learning  environment. 

Is  Online  Learning  Different? 

Why  is  it  necessary  to  explore  and  evaluate  the  issues  of  culture  and  motivation  in  online  learning 
environments,  as  distinguished  from  other  specific  environments?  After  all,  there  already  exists  persuasive 
evidence  that  culture  does  matter,  whether  in  the  classroom,  in  textbook  construction,  or  in  testing.  So  it  is 
worthwhile  to  ask  the  question:  Why  should  online  learning  need  to  be  investigated  separately? 

Although  research  does  not  generally  indicate  that  learning  outcomes  are  different  in  online  learning  as 
compared  to  other  learning  delivery  methods  (Saba,  2000),  Winn  and  Snyder  (1996)  noted  that  traditional 
theories  of  distance  education  evolved  while  behaviorist  models  were  prevalent;  thus  cognitive  psychology  and 
cognitive  science  have  been  incompletely  integrated  in  distance  education  theory  and  models. 

There  is  evidence  to  support  the  contention  that  online  learning  environments  impose  different  types  of 
cognitive  loads  on  the  learner.  This  was  first  proposed  by  Kozma,  who  argued  that  there  must  be  recognition  of 
the  cognitively  relevant  characteristics  of  media.  Kozma  differentiated  between  the  learner’s  internal  and  the 
external  environments,  specifying  that  the  learner  must  use  his  or  her  internal  cognitive  resources  to  extract 
information  from  the  external  environment  during  the  process  of  constructing  new  knowledge  (Ullmer,  1992). 

This  position  was  strengthened  by  recent  research  comparing  brain  activity  in  virtual  and  real 
environments  (Micropoulos,  2001).  In  this  exploratory  study,  participants’  EEG  readings  were  recorded  while 
executing  a simple  task  in  a virtual  environment  and  the  same  task  in  a real  environment,  and  the  two  readings 
were  compared.  Significant  differences  were  found  in  the  readings,  indicating  that  different  cognitive  processes 
were  being  used  when  in  the  virtual  environments. 


183 


Related  to  this  is  Prensky’s  (2001)  contention  that  “digital  natives”  actually  think  differently  from 
those  who  are  not  accustomed  to  using  digital  accessories  and  games.  He  argued  that  the  very  use  of  the  digital 
tools  has  modified  the  cognitive  processes  of  those  who  have  grown  up  with  them.  Other  research  on  such 
phenomena  as  split-attention  effects  (Mayer  & Moreno,  1998)  has  tended  to  support  similar  conclusions. 

If  it  is  true  that  the  cognitive  load  imposed  by  online  learning  differs  qualitatively  from  that  of  other 
learning  environments,  then  perceptions  and  reactions  that  are  intertwined  with  cognition  are  also  likely  to  vary. 
A review  of  how  culture  and  cognition  interact  will  illustrate  how  this  may  be  true. 

The  Relationship  Between  Culture  And  Cognition 

A sociocultural  view  of  cognitive  ability  first  appeared  in  the  early  1900s  (Sticht,  1994),  although  at 
that  point  it  was  almost  the  opposite  of  what  it  is  today.  A century  ago  scholars  believed  that  "primitive"  people 
had  primitive  cognition  patterns  such  that  the  cognition  patterns  and  the  cultures  reinforced  each  other  (Cole, 
Gay,  Glick  & Sharp,  1971),  making  members  of  those  cultures  unable  to  think  in  complex  or  “advanced”  ways. 
With  behaviorism,  however,  came  the  belief  that  cognition  is  essentially  the  same  across  cultures,  regardless  of 
cultural  norms  and  practices;  anthropologists  especially  held  that  position,  believing  that  people  varied  only  on 
cultural  practices. 

However,  behaviorism  fails  to  explain  why  different  cultures  develop  such  radically  different  practices 
if  their  cognitive  patterns  are  essentially  the  same.  The  concept  of  “World  View”  prevalent  in  the  1970s  and 
1980s  attempted  to  deal  with  this  issue  by  analyzing  the  “culturally  specific  cognition  of  a people”  and 
representing  it  “in  terms  of  a set  of  logically  interrelated  and  structurally  consistent  propositions  and  corollary 
statements  that  are  assumed  to  model  native  perception  and  thinking"  (Kearney,  1984,  p.  36).  Kearney  defined  a 
culture’s  worldview  as  a model  of  how  that  culture  looks  at  reality,  consisting  of  “...basic  assumptions  and 
images  that  provide  a more  or  less  coherent,  though  not  necessarily  accurate,  way  of  thinking  about  the  world.” 
(p.  41)  Kearney  further  noted  that  different  worldviews  developed  because  of  both  external  (i.e.,  environmental) 
and  internal  (i.e.,  cognitive)  reasons;  however,  this  stance  incompletely  addresses  the  question  of  interaction 
between  culture  and  cognition. 

The  sociocultural  understanding  of  cognition  has  regained  currency  recently,  as  social  constructivism 
and  contextualism  has  emerged,  according  to  Sticht  (1994);  Sticht  noted  that  this  approach  attempts  to  explain 
the  cognitive  development  of  humans  in  general  and  social  groups  (cultures)  as  well  as  individual  development. 

Contributing  to  and  fueling  this  re-emergence  was  a growing  interest  in  diversity,  and  thus  in  how  one 
culture  behaved  as  compared  to  another  (e.g.,  Dick  & Robinson,  1997).  Such  comparisons  often  took  the  form 
of  a litany  of  differences  that  were  largely  anecdotal  and  without  theoretical  basis,  but  they  did  attempt  to 
categorize  and  rationalize  cultural  differences.  Unfortunately,  they  also  had  the  tendency  to  reduce  cultures  to 
stereotypical  lists  of  characteristics. 

As  knowledge  regarding  different  aspects  of  cognition  was  expanded,  there  began  to  be  cultural 
analyses  that  emphasized  those  aspects.  For  example,  Griggs  and  Dunn  (1996)  considered  learning  styles  of 
Hispanics,  mentioning  their  "other-directedness"  which  conflicts  with  the  US  mainstream  individualism  and 
noting  that  Hispanics'  emphasis  on  cooperation  can  result  in  discomfort  with  the  competitiveness  of  the 
classroom.  A similar  study  was  produced  by  Chen  and  Stevenson  (1995)  who  looked  at  motivation  and 
mathematics  achievement  in  Asian-American,  Caucasian-American,  and  East  Asian  high  school  students. 

Cognition,  Learning  Theories,  and  Social  Contexts 

During  the  60’ s and  70s,  the  influence  of  Piaget  caused  learning  and  intelligence  to  be  seen  as  a 
progressive  process  involving  feedback  and  stages  of  cognitive  development.  The  individual  and  the 
characteristics  of  the  individual’s  mental  organization  was  at  the  center  of  Piagetian  theory,  and  social  issues 
were  at  best  secondary,  being  indicators  of  progress  rather  than  contributing  factors  (Light  & Perret-Clermont, 
1989). 

In  contrast,  Vygotsky  treated  cognitive  development  and  higher  mental  functions  as  primarily  a social- 
cultural  product,  with  cultural  knowledge  and  values  providing  the  basis  of  reasoning,  inferencing,  and 
interpreting  meanings.  Vygotsky  also  linked  culture  with  language  development,  and  language  with  learning, 
providing  an  additional  link  or  anchor  into  cultural  meaning  making  (Trueba,  1993). 

An  offshoot  of  Vygotskian  thought,  activity  theory  was  advanced  by  Luria,  Leont'ev,  and  Zinchenko.  It 
takes  as  its  main  focus  the  sociocultural  nature  of  intellectual  development,  according  to  Gauvain  (2001). 
Activity  theory  is  based  on  three  main  assumptions:  (1)  behavior  is  goal-directed  and  practical,  (2)  cognitive 
development  is  a product  of  social  and  cultural  history,  and  (3)  cognition  is  a socially  mediated  process. 
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Gauvain  commented  that,  because  activities  and  their  settings  are  created  by  the  participants  in  that  setting,  they 
reflect  the  group’s  assumptions,  resources,  and  goals.  “This  notion  transcends  the  boundary  between  the 
individual  and  the  social.  In  so  doing,  it  connects  the  interpsychological  plane,  that  is,  between  individuals,  and 
the  intrapsychological  plane,  that  is,  within  an  individual,  of  human  functioning  and  development.”  (p.  48)  Thus 
cognitive  development  is  the  means  by  which  the  individual  shapes  his/her  biological  capabilities  to  conform  to 
the  social  environment  in  which  the  individual  is  active.  But  Gauvain  cautioned  that  activity  theory  is  limited 
because  it  does  not  specify  which  social  processes  shape  intellectual  growth  or  connect  specific  features  of 
social  interaction  to  specific  facets  of  cognitive  growth. 

An  early  example  of  the  application  of  the  sociocultural  theory  of  cognition  is  found  in  the  study 
published  by  Cole  et  al  (1971).  It  is  an  exhaustively  detailed  ethnography  of  the  Kpelle  in  Liberia  that  sought  an 
explanation  of  why  Kpelle  children  have  so  much  trouble  with  Western -style  mathematics,  and  in  doing  so  it 
demonstrated  how  culture  and  thought  processes  are  intertwined.  The  researchers  found  significant  differences 
between  the  Kpelle  and  Americans  in  uses  of  taxonomies,  class  distinctions/heuristics,  memory  skills,  etc.  Their 
primary  conclusion  was 

...that  cultural  differences  in  cognition  reside  more  in  the  situations  to  which  particular  cognitive 
processes  are  applied  than  in  the  existence  of  a process  in  one  cultural  group  and  its  absence  in 
another.  Assuming  that  our  goal  is  to  provide  an  effective  education  for  everyone...,  our  task  must  be 
to  determine  the  conditions  under  which  various  processes  are  manifested  and  to  develop  techniques 
for  seeing  that  these  conditions  occur  in  the  appropriate  educational  setting,  (p.  233) 

Bandura  continued  to  focus  on  social  constructs,  in  particular  expanding  the  notion  of  self-efficacy  to 
include  the  concept  of  “collective  agency”  (Pajares,  2002).  This  is  defined  as  “a  group’s  shared  belief  in  its 
capability  to  attain  goals  and  accomplish  desired  tasks”  (Pajares,  2002,  Self-efficacy  Beliefs,  paragraph  7). 

Clearly,  current  learning  theory  has  progressively  emphasized  the  role  of  the  culture  in  the 
development  of  cognition  and  learning.  But  how  is  this  accomplished?  What  is  the  nature  of  the  interaction 
between  the  culture  and  the  developing  intellect  that  would  make  this  so? 

On  The  Social  Context  Of  Cognition 

At  the  organic  level,  it  is  assumed  that  all  “normal”  brains  function  roughly  in  the  same  way  within  a 
broad  range  in  terms  of  perceiving  and  conveying  data  (Carter,  1998).  That  data  is  then  processed  into 
information  and  stored;  this  “information  processing  approach”  (Anderson,  2000)  is  assumed  to  be  common  to 
all  “normal”  human  cognition.  But  an  important  corollary  of  these  concepts  is  that  the  brain  perceives  and 
processes  information  by  using  pathways  and  schemata  laid  down  by  previous  experience;  consequently,  each 
successive  cognitive  experience  is  progressively  more  affected  by  what  the  individual  has  perceived  and 
experienced  previously.  So  over  time,  perceptions  of  experiences  and  knowledge  provided  by  the  environment 
(including  the  cultural  context)  will  literally  change  the  flow  of  the  same  mental  processes  from  which  they 
emerged  (Valsiner,  1996;  Anderson,  2000;  Carter,  1998;  Sticht,  1994). 

Not  only  perception,  but  also  reasoning  is  strongly  influenced  by  culture.  Reasoning  depends  on 
schemata,  many  of  which  are  supplied  by  the  cultural  context  (Hutchins,  1980,  cited  by  D’ Andrade,  1989). 

D’ Andrade  concludes  that  when  differences  in  problem  solving  are  found  between  groups  of  people,  it  is  much 
more  likely  that  this  is  the  result  of  a difference  in  shared  cognitive  structures,  or  culture,  between  the  groups 
than  the  result  of  a genetic  difference  in  some  kind  of  general  reasoning  ability. 

These  perspectives  are  further  buttressed  by  the  theory  of  ecological  psychology,  which  argued  that  the 
mind  and  the  environment  must  be  treated  as  a unity  rather  than  separate  and  independent  entities  (Costall, 

1989;  Rosche,  1996).  Similarly,  the  theory  of  situated  learning  (Sticht,  1994;  Stein,  1998)  asserted  that  learning 
results  from  a social  process  involving  a variety  of  thought,  perception,  problem-solving,  and  interaction;  thus 
learning  is  not  separate  from  the  physical,  dynamic  world,  but  connected  to  it  through  complex  social 
environments.  Downes  (2004)  transferred  that  concept  to  online  environments,  commenting  on  the  importance 
of  social  interactions  in  learning;  he  especially  noted  its  importance  in  computer  supported  collaborative 
learning  and  touched  briefly  on  the  cultural  aspect  of  social  interactions. 

If,  indeed,  social  context  is  intimately  involved  with  the  development  of  cognition,  what  are  the  means 
by  which  it  leaves  its  mark,  and  on  what  aspects  of  cognition?  Culture  has  traditionally  been  defined  as  a stable 
set  of  norms,  beliefs,  and  behaviors;  however,  during  the  past  two  decades,  culture  has  been  seen  as  consisting 
of  knowledge  and  conceptual  structures  (Valsiner,  1996).  If  so,  how  do  those  structures  and  knowledge  impinge 
upon  and  affect  the  developing  mind  such  that  they  leave  a lasting  imprint? 

Gauvain  (2001,  1995)  examined  in  minute  detail  the  research  that  points  to  the  social  foundation  of 
developmental  cognition.  She  considered  the  sociocultural  context  of  development  to  provide  the  core  activities 
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through  which  children  are  exposed  to,  and  learn  about,  thinking;  she  believed  that  the  vast  majority  of  the 
cognitive  functions  that  children  develop  in  the  early  to  middle  years  of  childhood  are  connected  to  social 
experience  in  ways  that  are  both  intricate  and  interrelated. 

She  identified  three  subsystems  (Gauvain,  1995)  that  serve  as  a sociocultural  structure  within  which 
cognition  develops: 

• Cultural  activity  goals  and  values 

• Tools  and  materials  provided  by  the  culture  to  meet  the  goals  and  values 

• High-level  cultural  structures  (e.g.,  scripts,  routines,  and  rituals)  that  help  the  culture  implement 
the  goals  and  values  in  socially  organized  and  cohesive  ways 

These  subsystems  both  assist  and  constrain  the  cognitive  development  of  the  culture’s  members,  and  channel 
human  thinking  in  ways  appropriate  to  and  supportive  of  the  culture. 

It  may  seem  intuitively  obvious  that  children  are  taught  by  adults  in  a culture,  but  what  is  less  obvious 
is  that  those  adults  quickly  and  completely  pass  on  their  own  cultural  values  and  goals  to  the  children  (Rogoff, 
1989).  Thus,  children  are,  in  effect,  apprentices  to  their  culture,  and  learn  concepts,  e.g.,  amount,  number,  area, 
volume,  weight,  etc.,  that  exist  in  their  culture  because  they  are  useful  in  that  environment  (Light  & Perret- 
Clermont,  1989).  Gauvain  (2001)  referred  to  this  process  as  cognitive  socialization,  and  noted  that  it  emphasizes 
the  cultural  and  goal-directed  nature  of  these  interactions  as  well  as  requiring  the  learner  to  play  an  active  ro  le 
in  the  process.  By  linking  the  larger  sociocultural  context  of  cognition  with  the  individual  context  of  cognitive 
growth,  the  learner’s  mind  is  organized  and  shaped  “in  ways  that  are  suited  to  the  needs  and  aspirations  of  the 
community  in  which  growth  occurs.’’  (p  34). 

Higher  mental  functions  identified  by  Gauvain  and  indicated  by  research  also  to  be  socially  co- 
constructed processes  include: 

• Problem  solving  skills:  Transfer  of  cultural  knowledge  is  involved  in  terms  of  what  features  of  a 
problem  space  to  encode,  strategies  to  use,  and  knowledge  base  development.  Values  are  also 
transmitted  about  the  problem  domain  and  the  categories  of  thinking  that  problems  represent. 

• Memory:  both  content  and  process  are  socially  co-constructed  processes.  In  the  process  of  developing 
memory,  individuals  absorb  values  represented  as  memories  as  well  as  specific  strategies  for 
remembering. 

• Planning:  Social  context  is  involved  in  learning  how  to  plan  actions  in  order  to  reach  goals  and  how  to 
coordinate  plans  with  those  of  others. 

Gauvain  did  not  deny  that  the  individual  brings  intrinsic  capabilities  to  social  interactions  and  therefore 
to  the  developmental  process.  But  she  focused  on  the  research  that  supports  the  view  that  much  of  cognitive 
development  is  a shared  domain  between  the  individual  and  society.  If  her  view  is  correct,  there  should  be 
persuasive  evidence  that  adults  in  different  cultures  actually  have  differing  thought  patterns. 

Such  research  has  recently  been  published  by  Nisbett  (2003).  Basing  his  conclusions  on  a series  of 
experiments  conducted  by  himself  and  others  and  supporting  them  with  an  analysis  of  cultural  history,  he 
contended  that  East  Asians  and  Westerners  differ  in  terms  of  whether  they  perceive  the  world  holistically  or  as 
collections  of  objects,  their  conception  and  use  of  logic  and  categorization,  their  valuation  of  individualism 
versus  group  harmony,  their  use  and  understanding  of  causal  attribution,  their  inclination  to  apply  rules  to 
situations,  their  development  of  relationship  skills,  and  much  more.  His  research  tends  to  bear  out  Gauvain’ s 
assertions  about  cognitive  development,  and  shows  us  the  degree  to  which  the  typical  adult’s  cognitive  profile 
can  differ  from  culture  to  culture. 


Culturally  Defined  Value  Systems 

Gauvain,  Nisbett,  and  others  have  indicated  the  degree  to  which  the  sociocultural  context  affects 
cognition  and  how  social  goals  and  values  underlie  many  cognitive  processes.  But  that  begs  the  question:  to 
what  extent  do  value  systems  (and  therefore  goals)  reflect  cultural  identity?  This  is  an  important  question  in  the 
current  inquiry,  because  a distinct  correlation  between  value  clusters  and  culture  is  required  in  order  to  search 
for  the  effect  of  a given  culture  in  an  online  learning  environment,  or  to  differentiate  between  two  or  more 
cultures. 

Recent  analyses  of  culture  and  attitudes  in  teaching  indicate  that  values  do,  indeed,  matter.  For 
example,  Boufoy-Bastick  (2001)  noted  that  strategies  for  improving  computer-related  attitudes  and  beliefs  of 
young  Latino  students  are  needed  as  many  do  not  see  computers  as  being  relevant  in  either  their  careers  or  their 
personal  lives.  Ziegahn  (2001)  remarked  on  the  potential  variance  between  adult  education  teachers’  values  and 
those  of  their  students. 
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The  term  “values”  can  vary  somewhat  in  definition  from  scholar  to  scholar,  but  it  consistently  carries 
with  it  the  concept  of  normative  orientations,  of  preferred  or  even  obligatory  conduct  and  of  desirable  and 
undesirable  conditions  (Williams,  1979).  It  is  recognized  that  societies  (as  well  as  institutions)  have  specific 
value  priorities  or  hierarchies  (Rokeach,  1979;  Williams,  1979). 

Rokeach  (1979),  whose  quantitative  research  some  three  decades  ago  clearly  established  the  fact  that 
values  were  differently  prioritized  by  various  institutions  and  societies,  contended  that  values  were  organized 
into  value  systems  by  organizations  and  societies,  and  that  a main  determinant  of  values  is  one’s  culture.  Using 
a list  of  18  terminal  values  (i.e.,  ideal  end-states  of  existence)  and  18  instrumental  values  (i.e.,  ideal  modes  of 
behavior)  Rokeach  evaluated  the  value  systems  of  many  different  groups,  concluding  that  the  actual  number  of 
terminal  and  instrumental  values  that  an  individual  or  an  organization  has  is  fairly  small. 

But  it  was  Hofstede’s  astonishingly  wide  data-gathering  work  that  led  to  a more  comprehensive  value 
structure  across  countries.  In  a work  first  published  in  1980  and  augmented  in  a second  edition  (2001),  Hofstede 
detailed  the  results  of  a series  of  surveys  of  cultural  values  conducted  between  1966  and  1978,  then  added  to  in 
1985-1995.  The  surveys  included  some  116,000  questionnaires  in  72  countries,  using  20  languages;  respondents 
were  members  of  the  IBM  workforce.  Later,  even  more  data  from  non-IBM  respondents  was  added. 

Using  the  data  produced,  Hofstede  and  his  associates  constructed  a database  suitable  for  statistical 
analysis.  Extensive  statistical  analysis  and  data  reduction  techniques  revealed  a structure  of  five  axial  data 
values  (or  clusters  of  values)  on  which  the  national  cultures  surveyed  differed  from  one  another.  Significantly, 
countries  seemed  to  group  together  on  each  of  the  structural  axes  in  ways  that  suggested  a degree  of  cultural 
consistency  (e.g.,  Hispanic  countries  tend  to  group  near  each  other  on  most  of  the  axes). 

Hofstede’s  five  cultural  dimensions  are: 

• Power  distance:  the  degree  to  which  the  less  powerful  members  of  the  society  accept  and  agree  that 
power  is  distributed  unequally;  the  acceptance  of  power  inequality  in  the  society. 

• Uncertainty  avoidance:  a measure  of  how  comfortable  or  uncomfortable  members  of  a culture  are  in 
unstructured  situations;  how  much  the  society  accepts  the  novel/surprising/unknown  versus  how  much 
it  tries  to  control  it.  This  concept  is  not  analogous  to  risk  avoidance;  rather  it  is  a tolerance  for 
ambiguity  or  uncertainty. 

• Individualism/collectivism:  the  balance  in  the  society  between  the  requirement  that  individuals  take 
care  of  themselves  versus  integrating  into  groups;  the  degree  to  which  social  referencing  is 
encouraged;  whether  the  individual  identifies  strongly  with  a group  and  is  indivisible  from  it,  or 
whether  the  individual  primarily  sees  him/herself  in  self-defined  terms,  separate  from  group  identity. 

• Masculinity/femininity:  the  width  of  the  divide  between  gender-based  roles;  the  degree  to  which 
biological  differences  are  expected  to  be  reflected  in  social  and  emotional  roles. 

• Long-term/short-term  orientation:  the  degree  to  which  members  of  a society  are  expected  to  be  able 
to  accept  delayed  gratification  of  material,  social,  and  emotional  needs;  persistence  and  thrift  are 
aspects  of  this  continuum. 

Hofstede  offered  the  first  comprehensive,  data-derived  model  of  cultural  values.  He  defined  the  model  and  its 
components  in  ways  that  are  usable  and  coherent,  consistent  with  previous  research  in  the  field,  and  statistically 
defensible. 


Motivation  And  Persistence 

It  is  clear  from  the  foregoing  discussion  that  values  are  culturally  anchored,  and  deeply  intertwined 
with  cognition  and  therefore  with  learning.  But  what  is  the  connection  of  values,  cognition,  motivation,  and 
persistence? 

Thought  and  theory  regarding  motivation  in  general  has  undergone  much  development  in  the  last 
century,  especially  since  McClelland  (1961)  and  Atkinson  (1964)  respectively  introduced  their  works  on 
achievement  and  expectancy  theories.  Atkinson  examined  such  factors  as  anxiety  regarding  failure,  expectancy 
of  success,  and  need  for  achievement,  but  considered  these  to  be  individual  characteristics  only.  McClelland, 
however,  saw  an  effect  of  culture,  at  least  obliquely,  by  considering  social  practices  such  as  methods  of 
childrearing  that  he  felt  contributed  to  the  individual’s  need  for  achievement. 

A few  years  later,  Raynor  (1967)  addressed  what  was  essentially  a weakness  of  expectancy  theory — 
that  it  was  concerned  only  with  the  expectations  of  success  and  failure  in  the  activity  being  observed.  In  his 
model  of  motivation,  Raynor  integrated  the  importance  of  long-term  goals  and  expectancies,  noting  that 
anticipated  future  consequences  of  present  behavior  differentially  affects  individuals,  depending  on  the  strength 
of  their  achievement-related  motives. 
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Maehr  (1974)  did  early  studies  specifically  pointed  at  cultural  aspects  of  motivation,  examining 
logically  various  constructs  that  might  represent  the  interaction  among  culture,  personality,  and  motivation.  He 
observed  the  effect  that  social  roles  may  have  on  certain  types  of  behavior  and  recommended  that  more  should 
be  done  to  analyze  the  influence  of  social  norms  on  motivational  behavior.  Maehr  noted  that  culturally  derived 
beliefs  about  ends  (terminal  values)  and  means  (instrumental  values)  typically  played  little  or  no  role  in  then- 
current  achievement  motivation,  but  felt  that  they  should. 

Keller’s  (1983;  1987)  ARCS  model  brought  together  the  above  themes  in  motivational  thought  plus 
many  more,  including  those  of  such  theorists  as  Weiner,  de  Charms,  Rotter  and  Bandura.  ARCS,  which  stands 
for  Attention,  Relevance,  Confidence,  and  Satisfaction,  provides  a systematic  approach  to  incorporating 
motivational  tactics  into  instruction.  The  learners’  goal  orientations  are  implicit  in  the  Relevance  aspect  of  the 
model,  and  their  perceived  success  in  having  met  those  goals  is  included  in  the  Satisfaction  phase. 

Motivational  Systems  Theory 

Still,  motivation  as  a field  did  not  have  a single,  unifying  theory  until  Ford’s  (1992)  work.  Anchored 
within  a comprehensive  theory  of  human  functioning  called  the  Living  Systems  Framework,  Ford’s 
Motivational  Systems  Theory  (MST)  provides  a complete  formulation  of  the  basic  characteristics  and 
interactions  of  motivation  and  competence  development. 

Ford  defines  motivational  processes  as  having  three  primary  characteristics: 

• They  are  qualities  of  the  person  rather  than  properties  of  the  context. 

• They  are  future  -oriented  rather  than  being  focused  on  the  past  or  present. 

• They  are  evaluative  rather  than  instrumental  in  character. 

So  motivation  can  be  facilitated  or  constrained,  but  not  imposed  on  a learner  under  Ford’s  theory,  as  it  is 
entirely  internal  to  the  individual. 

Within  MST,  motivation  is  defined  as  "the  organized  patterning  of  an  individual's  personal  goals, 
emotions,  and  personal  agency  beliefs.”  (p.  78)  Thus  the  concept  can  be  restated: 

Motivation  = Goals  x Emotions  x Personal  Agency  Beliefs 

Ford  assigned  goals  a leadership  role  in  motivation,  specifying  that  both  cognitive  and  emotional 
evaluations  underlie  the  formation  of  new  goals.  He  contended  that  a large  portion  of  one’s  feelings  of 
satisfaction  and  frustration  can  be  traced  to  the  organizing  aspect  of  one’s  goals  , and  that  the  most  motivating 
activities  in  life  will  be  those  that  involve  the  simultaneous  pursuit  and  attainment  of  multiple  personal  goals. 
Goals  include  both  content  (representing  the  consequences  to  be  achieved  or  avoided)  and  process  (directing  the 
other  components  and  capabilities  of  the  person  to  try  to  produce  those  consequences). 

Thus  Ford’s  theory  fits  neatly  into  the  space  provided  by  Rokeach  and  Hofstede  on  one  hand,  and 
Gauvain  and  Nisbett  on  the  other.  As  discussed  earlier,  Rokeach  and  Hofstede  made  clear  that  goals  and  the 
values  that  support  them  are  heavily  influenced  by  one’s  national  culture,  and  Gauvain  and  Nisbett  showed  in 
exhaustive  detail  the  extent  to  which  culture  is  seen  to  affect  the  individual’s  cognitive  processes.  This 
interlinking  of  values,  goals,  cognition,  and  culture  is  at  the  heart  of  the  theoretical  connections  being  proposed 
by  this  paper.  A schematic  view  of  these  linkages  is  shown  in  Figure  1. 

In  fact,  Gauvain  (2001)  nibbled  around  the  edges  of  this  when  she  remarked  that  children's  learning 
and  involvement  with  their  community  show  "patterns  reflecting  both  short-  and  long-term  goals  and  values  of 
the  communities.  These  variations  would  be  expected  to  lead  to  differences  in  what  children  learn  to  think  about 
and  how  they  learn  to  think."  (p.  40) 

The  other  two  components  of  MST  also  show  unmistakable  connections  with  the  individual’s  culture. 
Personal  Agency  Beliefs  are  evaluative  thoughts  (and  therefore  anchored  in  cognition)  that  compare  desired  and 
anticipated  consequences;  but  they  have  no  meaning  or  functional  significance  if  the  goal  they  support  is 
without  value  to  the  individual.  They  are  seen  as  being  more  fundamental  than  the  actual  skills  and 
circumstances  they  represent,  because  they  can  encourage  people  to  open  opportunities  and  acquire  capabilities 
that  they  do  not  yet  possess;  thus  they  serve  as  a potentiating  force.  Ford  notes  that  they  play  “a  particularly 
crucial  role  in  situations  that  are  of  the  greatest  developmental  significance — those  involving  challenging  but 
attainable  goals.”  (p  124).  Note  that  Bandura’s  notion  of  “collective  agency”  (described  above)  is  an  expanded 
version  of  this  concept  that  illustrates  the  cultural  connections  even  more  clearly. 

Likewise,  emotions  (in  older  children  and  adults)  are  generally  “activated  by  cognitive  evaluations 
pertaining  to  current  or  potential  concerns  in  real  or  imagined  circumstances”  (p.  143)  according  to  MST. 
Regardless  of  whether  the  emotions  are  conscious  or  not,  they  may  involve  habitual  patterns.  Both  in  terms  of 
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their  cognitive  component  and  in  terms  of  their  connection  with  ingrained  patterns,  emotions  clearly  have 
connections  with  culture. 

Emotions  have  long  been  considered  to  have  a clear  interaction  with  online  learning  as  well.  Malone 
and  Lepper  (1987;  also  Lepper  and  Malone,  1987)  wrote  extensively  on  the  importance  of  using  “motivational 
embellishments”  to  create  a sense  of  fun,  challenge,  curiosity,  and  fantasy  in  online  learning  in  order  to  engage 
learners  and  enhance  their  intrinsic  motivation. 

Finally,  it  should  be  noted  that  MST  defines  competence  as  “the  attainment  of  relevant  goals  in 
specified  environments,  using  appropriate  means  and  resulting  in  positive  developmental  outcomes”  (p.  67). 
Accordingly,  the  concept  of  competence  is  also  intimately  connected  with  cultural  issues  as  regards  both  goals 
and  context. 

Thus  every  aspect  of  Ford’s  theory,  when  juxtaposed  with  other  relevant  models  and  theories,  is 
permeated  by  the  effects  of  the  individual’s  cultural  background  and  the  effects  that  background  unavoidably 
imposes.  When  these  interactions  are  mediated  by  an  online  environment,  the  motivational  outcome  may  be 
different  than  that  in  other  environments.  And  because  we  have  seen  both  theoretical  and  research-based 
evidence  that  online  environments  differ  from  other  learning  environments  in  terms  of  their  cognitive  load  and 
effects  on  the  learner,  it  is  reasonable  to  suggest  that  the  role  of  culture  on  motivational  outcomes  in  online 
environments  is  worth  studying. 


Conclusion 

Cognition,  cultural  goals,  and  values,  motivation. . . the  pieces  are  all  there  for  online  learning  and 
instructional  design  experts  to  find.  The  problem  is  that  they  are  scattered  across  disparate  disciplines: 
anthropology,  psychology,  sociology,  and  education.  Given  the  current  state  of  relevant  knowledge  and  theory, 
it  would  be  illogical  that  there  might  not  be  an  effect — or  a range  of  effects — on  learner  motivation  and 
persistence,  rooted  in  the  interplay  between  the  online  learning  environment  and  learner  culture.  It  only  remains 
to  find  what  those  effects  might  be  so  that  we  can  allow  and  plan  for  them,  and  in  so  doing,  provide  added 
means  by  which  all  learners  can  advance  equally  on  the  path  offered  by  online  learning. 

Unfortunately,  so  far  research  in  this  area  is  very  sparse.  (Collis  & Remmers,  1997;  Collis,  1999; 
Thomas,  2002).  As  learning  becomes  increasingly  globalized  and  as  academic  and  other  organizations  move 
forward  with  plans  to  encourage  learners  from  other  cultures  to  join  them,  it  would  be  a shame  if  our  own 
ethnocentrism  prevented  us  from  understanding  that  other  cultures  may  react  differently  than  we  do  to  the 
virtual  learning  environments  we  have  implemented.  Only  by  testing  and  engaging  in  continued  research  can  we 
evaluate  whether  the  learning  structures  we  have  designed  will  provide  equal  access  and  opportunities  for 
success  to  all  learners. 
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Abstract 

There  is  a wealth  of  information  available  on  the  World  Wide  Web  that  can  assist  pre-ser\’ice 
teachers  in  their  course  studies.  Yet  obserx’ation  of  students  in  a technology  integration  class 
indicated  that  students  were  not  able  to  find  resources  efficiently  or  reliably.  The  purpose  of 
this  study  was  to  establish  a baseline  of  what  undergraduate,  pre-sen’ice  teachers  know  about 
search  engines  and  searching  the  Web  prior  to  beginning  a course  on  integrating  technology 
into  the  classroom.  A total  of  355  undergraduate,  pre-sen’ice  teachers  over  three  semesters 
participated  in  the  study.  The  results  indicate  low  declarative,  syntactic  and  semantic 
knowledge.  Implications  for  course  and  program  revisions  based  on  the  results  are 
addressed. 

Academic  programs  are  continuously  under  evaluation  and  revision.  Whether  through  a formal  process 
from  an  outside  agency,  or  the  individual  reflections  of  an  instructor,  these  reviews  seek  to  determine  the 
effectiveness  and  value  of  a specific  course  or  program  in  terms  of  the  validity  of  objectives,  relevancy  and 
sequence  of  content,  relation  between  assumed  prior  knowledge  and  current  course  content,  and  achievement  of 
specific  goals.  In  an  internal  review  of  a technology  integration  course  at  a southeastern  university,  the 
researchers  found  that  faculty  in  the  college  of  education  assumed  students  at  the  university  knew  how  to  use 
the  Internet  as  a source  of  information  to  support  their  academic  studies.  Yet  the  researchers  observed  many 
students  had  difficulty  locating  information  relevant  to  their  courses  and  assignments.  The  researchers  also 
observed  students  spent  a great  deal  of  time  on  unsuccessful  searchers;  few  students  were  able  to  locate  relevant 
resources  quickly.  The  purpose  of  this  study  was  to  establish  a baseline  of  what  undergraduate,  pre  -service 
teachers  know  about  search  engines  and  search  strategies.  The  results  would  then  be  used  to  determine  if  any 
changes  in  the  curriculum  were  warranted. 


Literature  Review 

Many  people  turn  to  the  World  Wide  Web  as  a source  of  information.  Whether  searching  for 
information  on  travel,  health,  entertainment,  or  academic  resources,  the  Web  has  many  attractive  qualities:  it  is 
easy  to  access,  it  is  "open"  24  hours  a day,  seven  days  a week,  there  is  no  need  to  venture  out  in  the  cold  or  rain, 
and  there  is  privacy  as  people  search  for  the  information  they  need.  Yet  research  among  the  general  public  has 
shown  searchers  looking  for  information  on  the  World  Wide  Web  have  a difficult  time  developing  search 
queries  and  using  a search  engine  (Chen,  Houston,  Sewell  & Schatz,  1998;  Lazonder,  Biemans  & Wopereis, 
2000).  Searches  tend  to  be  simple  (Spink,  Bateman  & Jansen,  1999)  and  Boolean  operators  are  used 
infrequently  and  incorrectly  (Jansen,  Spink  & Saracevic,  2000).  Although  time  spent  searching  the  Web  is  high 
(Sullivan  reports  31%  of  Internet  users  utilize  a search  engine  two  to  three  times  per  week  [Sullivan,  2001]), 
frustration  levels  are  also  high  with  71%  of  respondents  reporting  they  get  frustrated  when  searching  for 
information  on  the  Internet  (Sullivan,  2002).  Frustration  and  poor  search  skills  rarely  lead  to  positive  results.  It 
is  possible,  though,  that  college  students  may  have  more  experience  with  computers,  and  thus  may  be  more 
successful  at  web  searching  than  the  general  population. 

A 2001  survey  among  U.S.  college  students  who  use  the  Internet  for  school-related  assignments  found 
70%  of  respondents  reported  they  were  successful  in  finding  what  they  seek  most  of  the  time  (OCLC,  2002). 
These  students  have  confidence  in  their  ability  to  locate  information  for  their  assignments  and  they  find  the 
Web  easy  to  use.  The  study,  however,  only  polled  those  already  using  the  Internet.  Another  survey  conducted  in 
2001,  found  91  percent  of  students  rated  themselves  competent  in  accessing  information  on  the  Internet  (Osika 
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and  Sharp,  2002).  Do  all  college  students  report  such  success?  If  so,  is  this  self-reported  data  true  in  practice? 
Specific  to  the  review  of  this  course,  are  pre -service  teachers  adept  at  the  skills  that  will  help  them  find 
information  on  the  Internet? 

The  purpose  of  this  study  was  to  establish  a baseline  of  what  undergraduate,  pre -service  teachers  know 
about  search  engines  and  searching  the  Web  prior  to  beginning  a course  on  integrating  technology  into  the 
classroom.  The  following  research  questions  were  addressed: 

• What  factual  knowledge  do  these  students  have  about  search  engines? 

• When  given  questions,  how  do  these  students  structure  a search  string? 

• Can  these  students  describe  how  a search  engine  operates? 

• Does  knowledge  of  search  engines  vary  with  the  amount  of  time  a person  spends  on  the  Web? 

• Does  knowledge  of  search  engines  vary  with  the  age  of  the  student? 

The  study  examined  three  types  of  knowledge  associated  with  web  searching:  declarative  knowledge, 
syntactic  knowledge,  and  semantic  knowledge  (Colaric,  2002).  Declarative  knowledge  (Anderson,  1982)  refers 
to  understanding  facts;  in  this  case,  facts  about  search  engines.  Syntactic  knowledge  refers  to  knowledge  of  the 
language  units  and  rules  when  working  with  a computer  system  (Mayer,  1992);  in  this  case,  how  to  structure  a 
search  query  using  terminology  the  search  engine  can  interpret  correctly.  Semantic  knowledge  refers  to  the 
user's  understanding  of  the  major  locations,  objects,  and  actions  inside  a computer  system  (Bayman  & Mayer, 
1988;  Mayer,  1989). 


Method 


Participants 

Potential  participants  were  undergraduate  pre -service  teachers  at  a research  university  in  North 
Carolina.  All  students  enrolled  in  a pre  -service  teaching  course  on  integrating  technology  in  the  classroom 
during  three  semesters  (spring  2002,  fall  2003,  and  spring  2004)  were  invited  to  participate.  The  technology 
integration  course  is  required  for  College  of  Education  students  and  is  usually  taken  during  the  students'  junior 
year.  Completion  of  the  surveys  was  voluntary  and  no  incentive  was  provided  for  completion. 

Instrument 

The  survey  instrument  was  developed  previously  by  one  of  the  researchers  (Colaric,  2002).  Eighteen 
questions  were  used  to  gather  information  in  four  areas:  personal  information,  declarative  knowledge  of  search 
engines,  syntactic  knowledge  of  search  engines,  and  semantic  knowledge  of  search  engines.  The  questions 
included  short  answer  and  multiple  choice.  Eight  questions  related  to  participants'  personal  information:  age, 
gender,  number  of  semesters  completed  at  the  university,  major  field  of  study,  minor  field  of  study,  whether 
he/she  owns  a computer,  approximate  number  of  hours  per  day  searching  the  Web,  and  approximate  number  of 
hours  per  day  sending/receiving  email.  Five  questions  related  to  declarative  knowledge  of  search  engines. 
Participants  were  asked  to  answer  questions  about  whether  or  not  all  search  engines  work  the  same  way, 
whether  search  engines  look  at  all  web  sites  on  the  WWW,  the  difference  in  the  amount  of  results  obtained  by 
using  AND  and  OR,  the  name  of  the  program  used  by  search  engines  to  gather  web  sites,  and  whether  the 
search  term  used  needs  to  match  the  engine's  index  in  order  for  a site  to  be  returned.  All  questions  included  an 
option  of  "I  don't  know".  Three  questions  asked  participants  to  write  down  what  he/she  would  normally  type 
into  a search  engine  when  looking  for  information  on  a given  topic  (syntactic  knowledge).  An  example  is: 
"Suppose  you  want  to  find  web  sites  that  describe  the  naval  battles  that  took  place  during  the  Napoleonic  War. 
What  would  you  type  into  a search  engine?”  Two  questions  asked  the  participants  to  describe  what  a search 
engine  would  do  when  given  a particular  search  query.  Both  queries  contained  information  grouped  in 
parentheses,  as  well  as  use  of  the  Boolean  operators  AND  and  OR.  These  two  questions  assessed  the 
participants'  semantic  knowledge  of  a search  engine  by  asking  them  to  describe  what  goes  on  inside  the  system 
when  a command  is  executed. 

Data  Collection  and  Analysis 

The  materials  were  administered  in  the  pre -service  teaching  course  on  integrating  technology  during  its 
normal  time  and  day  in  the  first  or  second  week  of  classes.  Surveys  were  coded  by  the  researchers  and  results 
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were  analyzed  using  SPSS  (Statistical  Package  for  the  Social  Sciences).  Demographic  data  was  entered  as  the 
participants  recorded;  groupings  were  also  established  for  age  (traditional  undergraduate  age  of  19  to  22  and  age 
23+)  and  use  of  the  Internet  (both  Web  searching  and  email;  "low"  group  of  less  than  one  hour,  "medium" 
group  of  1 to  2.5  hours,  and  "medium-high"  group  of  3 to  4.5  hours,  and  "high"  group  of  5+  hours). 

Questions  for  the  declarative  knowledge  section  were  multiple  choice  with  answers  coded  and  entered 
into  SPSS.  For  the  syntactic  knowledge  section,  a numeric  score  for  each  answer  was  recorded  based  on  three 
categories:  accuracy  of  concepts  identified,  inclusion  of  variable  concepts,  and  accuracy  of  Boolean  expression. 
For  the  semantic  knowledge  section,  a numeric  score  for  each  answer  was  recorded  based  on  six  categories: 
understanding  of  OR  as  a join,  understanding  of  AND  as  an  intersect,  inclusion  of  all  terms  from  the  query, 
understanding  that  the  search  engine  is  querying  a database  set,  understanding  of  searching  for  a literal  string  of 
characters,  and  understanding  that  all  search  engines  operate  in  a unique  manner  (Colaric,  2002). 

Results 


Demographic  Results 

A total  of  355  students  completed  the  survey  over  the  three  semesters.  Participants  ranged  in  age  from 
19  to  57  with  a mean  age  of  23  (median  age  was  21).  Eighty-three  percent  of  the  participants  were  female; 
seventeen  percent  were  male.  Semesters  completed  ranged  from  one  to  sixteen;  the  mean  number  of  semesters 
completed  was  8.5.  Ninety  percent  of  participants  own  their  own  computers.  Fifty  percent  of  the  participants 
were  Elementary  Education  majors;  all  participants  were  studying  in  some  area  of  teacher  education. 
Participants  reported  spending  an  average  of  1.5  hours  per  day  searching  the  Web  (range  of  0 to  10)  and  1 hour 
per  day  on  email  (range  of  0 to  8). 

Comparisons  between  the  different  semesters  were  tested  to  determine  if  any  variations  existed;  the 
groups  were  found  to  be  similar  in  all  areas.  Cross  tabulations  were  run  to  determine  if  students  of  traditional 
college  age  (19-22;  n = 270)  spent  more  time  on  email  or  searching  the  Web  than  students  of  non-traditional  age 
(23+;  n = 84).  No  significant  differences  were  found. 

Declarative  Knowledge 

The  participants  in  the  study  appeared  to  have  some  prior  factual  knowledge  of  search  engines.  Most 
(n  = 355;  64%)  understood  search  engines  operate  differently  from  each  other.  Of  concern  are  the  14%  (n  = 49) 
of  participants  who  thought  search  engines  were  all  the  same  and  22%  (n  = 80)  participants  who  did  not  know  if 
search  engines  were  all  the  same. 

A fairly  high  number  of  participants  believe  search  engines  peruse  all  sites  on  the  Web  (n  = 143;  40%) 
while  a number  of  participants  (n  = 73;  21%)  are  not  sure  about  this  idea.  Forty-seven  percent  of  participants  (n 
= 168)  understood  that  terms  typed  into  a search  engine  need  to  match  the  indexed  sites  of  the  engine  in  order  to 
be  returned.  Thirty-three  percent  did  not  know  (n  = 115)  and  20%  thought  this  was  false  (n  = 7 1).  Less  than  half 
of  participants  (n  = 162;  46%)  understood  the  Boolean  operator  OR  retrieves  more  results  than  the  operator 
AND.  Twenty-five  percent  (n  = 89)  thought  OR  retrieved  less  results;  23%  did  not  know  (n  = 81)  and  6% 
thought  OR  retrieved  the  same  amount  as  AND  (n  = 23). 

There  were  no  statistically  significant  differences  between  the  age  groups  (19-22;  n = 270  and  23+;  n = 
84)  in  relation  to  declarative  knowledge.  Nor  were  there  statistically  significant  differences  in  the  amount  of 
time  spent  searching  the  Web  or  the  amount  of  time  spent  on  email  in  relation  to  declarative  knowledge. 

Syntactic  Knowledge 

When  asked  to  construct  a search  string  given  a particular  topic,  participants  tended  to  construct  very 
simple  queries  with  a mean  of  2.9  terms  per  query.  Scores  for  syntactic  knowledge  could  range  from  0 to  18;  the 
mean  score  was  5.36.  Twenty-five  percent  of  participants  used  AND  correctly  in  their  search  string;  1%  used 
AND  incorrectly.  One  percent  of  participants  used  OR  in  any  of  the  three  search  strings.  Twenty-seven  percent 
of  participants  included  stop  words  in  their  queries  (common  words  which  are  ignored  by  some  search  engines 
or  result  in  a high  number  of  listings  for  other  engines).  Three  percent  of  respondents  included  words  not 
directly  used  in  the  search  question;  in  all  cases  the  words  used  were  appropriate  synonyms  for  the  terms  in  the 
search  question.  Six  percent  of  participants  used  phrasing  with  1%  of  those  using  it  incorrectly.  Truncation, 
NOT,  and  search  modifiers  were  not  used  by  any  participants. 

There  were  no  statistically  significant  differences  between  the  age  groups  (19-22;  n = 270  and  23+;  n = 
84)  in  relation  to  syntactic  knowledge.  Nor  were  there  statistically  significant  differences  in  the  amount  of  time 
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spent  searching  the  Web  or  the  amount  of  time  spent  on  email  in  relation  to  syntactic  knowledge. 

Semantic  Knowledge 

Participants  were  generally  not  successful  in  describing  their  semantic  knowledge,  scoring  a group 
mean  of  .47  points  out  of  a possible  12  (standard  deviation  = 0.96;  range  = 1,5).  Two  hundred  and  seventy-two 
participants  (77%)  received  no  points  for  this  section.  Participants  who  did  respond  were  slightly  more  likely  to 
include  a description  of  AND  as  an  intersect  (15%)  than  include  OR  as  ajoin  (10%).  Most  participants  (81%) 
did  not  include  all  of  the  terms  included  in  the  question,  choosing  instead  to  describe  what  the  engine  would  do 
with  just  one  or  two  terms. 

There  were  no  statistically  significant  differences  between  the  age  groups  (19-22;  n = 270  and  23+;  n = 
84)  in  relation  to  semantic  knowledge.  Nor  were  there  statistically  significant  differences  in  the  amount  of  time 
spent  searching  the  Web  or  the  amount  of  time  spent  on  email  in  relation  to  semantic  knowledge. 

Discussion 

Evaluation  and  revision  are  necessary  in  all  successful  academic  programs;  however,  the  success  of 
programs  that  integrate  technology  goals  and  objectives  hinge  on  the  evolution  of  new  technological 
innovations  as  well  as  student’s  technical  skill  subsets.  As  program,  accreditation  and  state  standards  are  revised 
to  correlate  with  the  demands  of  the  ‘technology  age’  it  is  imperative  that  the  courses  that  support  technology 
integration  for  educators  rise  to  meet  and  exceed  the  needs  of  the  populations  that  they  serve.  The  purpose  of 
this  study  was  to  establish  a baseline  of  what  undergraduate,  pre -service  teachers  know  about  search  engines 
and  searching  the  Web  in  an  effort  to  evaluate  the  course  curriculum  in  place  and  determine  if  changes  are 
necessary  to  effectively  meet  the  goals  and  objectives  of  the  course.  A current  review  of  the  literature  as  well  as 
an  informal  review  of  the  course  indicated  that  students  may  not  have  the  prerequisite  skills  needed  to  engage 
effectively  in  the  Web  searching  skills  necessary  for  the  planned  course  content. 

Confirming  instructor  suspicions,  a significant  proportion  of  students,  36%,  surveyed  indicated  that 
they  did  not  have  the  declarative  knowledge  necessary  to  effectively  use  search  engines.  These  students 
believed  that  search  engines  did  not  differ  in  function  or  were  not  aware  of  their  function.  The  results  from  the 
survey  further  showed  that  most  students  had  misconceptions  regarding  Boolean  operators,  the  construction  of 
search  strings,  and  were  generally  not  successful  in  describing  their  semantic  knowledge  of  search  engines. 
These  results  corresponded  with  the  suspicions  of  the  course  instructors  noted  in  informal  reviews  and 
evaluations  of  the  course  but  contrasted  with  generally  held  assumptions  of  student  knowledge  of  web  searching 
held  by  faculty  in  the  College  of  Education  at  large.  Student  time  on  the  web  and  demographic  variations  did 
not  demonstrate  a significantly  statistical  difference  in  semantic,  systemic  or  declarative  knowledge  which 
defeats  the  notion  that  time  on  task  correlates  with  efficient  searching.  Instead,  time  on  the  Web  perpetuates 
poor  searching.  These  results  support  the  need  for  course  revisions  in  curriculum  programs  that  integrate  Web 
technology,  particularly  the  Educational  Technology  course  where  the  survey  was  conducted,  by  establishing  a 
baseline  of  student  knowledge.  Without  an  understanding  of  students’  prior  knowledge,  integration  of  new 
knowledge  is  bound  to  be  less  successful.  In  addition,  course  revisions  will  focus  on  a curriculum  alignment 
that  works  with  a triangulated  approach  to  search  engines  and  Web  searching.  Units  of  curriculum  will  focus  on 
evaluating  student’s  prior  knowledge  as  well  as  incorporating  declarative,  semantic  and  systemic  approaches  to 
instruction  of  search  techniques  as  a development  of  baseline  skills  necessary  to  achieve  the  goals  and 
objectives  necessary  for  program,  accreditation  and  state  standards. 

Sound  design  principles  require  that  course  developers  as  well  as  instructors  evaluate  the  effectiveness 
of  course  materials,  curriculum  and  objectives.  In  a technology  based  pre -service  teacher  course,  evaluations 
and  curriculum  revisions  must  reflect  the  innovations  presented  in  technology  as  well  as  in  schools.  The  results 
from  this  survey  suggest  that  curriculum  revisions  include  professional  development  with  in  the  College  of 
Education,  development  of  CAI  tutorials  for  student  instructional  supplement,  implementation  of  modules 
within  the  course  content  focusing  on  the  online  research  and  methods. 

The  use  of  technology  in  a College  of  Education  can  not  be  narrowed  down  to  one  specific  course. 
Effective  instruction  requires  that  students  are  given  the  opportunity  to  view  various  methods  of  integration, 
modeling  and  repetition  in  order  for  retention  and  meaningful  learning  to  take  place.  In  order  to  facilitate  and 
model  appropriate  schemas  of  technology  integration,  staff  development  for  all  College  of  Education  Faculty 
will  be  developed  and  made  available.  The  development  module  will  contain  training  on  Internet  research 
techniques,  Boolean  operations,  and  technology  integration  methods  for  University  faculty  that  correlate  with 
ISTE’s  NET-S  standards.  These  development  modules  will  offer  faculty  the  opportunity,  time  and  information 
necessary  for  implementation  within  their  course  curriculums. 
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Development  of  CAI  tutorials  as  well  as  the  implementation  of  curriculum  modules  within  the 
technology  service  course  have  also  been  recommended  based  on  the  survey  results.  CAI  tutorials  will  be 
developed  and  used  as  course  supplements  for  College  of  Education  faculty  as  well  as  faculty  teaching  the  pre- 
service  technology  course.  These  tutorials  in  conjunction  with  the  addition  of  a curriculum  module  of  web 
research  techniques  will  serve  as  the  curriculum  revisions  necessary  to  meet  the  needs  of  both  the  students  and 
faculty  within  the  pre -service  technology  course.  Faculty  will  continue  to  re-evaluate  course  curriculum  and 
student  needs  by  developing  a focused  evaluation  instrument  designed  to  measure  the  effectiveness  of 
instruction  based  on  the  three  areas  of  knowledge.  Declarative,  Syntactic  and  Semantic,  and  their  effective 
outcomes  on  meaningful  learning. 

A triangulated  approach  to  further  research  is  recommend  by  the  researchers  to  determine  the  global 
impact  of  the  relationship  between  the  declarative,  syntactic  and  semantic  knowledge  relationships  effect  on 
instruction  and  learning.  In  order  to  develop  CAI  training  and  curriculum  modules  that  are  instructionally 
effective  investigation  will  focus  on  the  relationships  between  meaningful  learning  and  the  three  knowledge 
domains.  Further  research  will  include  the  development  of  a pre-test/post-test  instrument  designed  to  evaluate 
the  revision  changes  that  will  be  implemented  in  the  technology  course  to  gauge  knowledge  acquisition 
changes.  These  results  will  assist  in  evaluating  if  student  needs  and  course  objectives  are  being  met  through  the 
implementations  made  through  the  suggestions  of  this  research.  Lateral  entry  teachers  as  well  as  teachers  in  the 
k-12  field  will  also  be  included  in  continued  research  efforts  to  determine  the  effects  of  experience  and  effective 
implementation  of  web  searching  techniques  and  prior  knowledge. 
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Abstract 

Retention  of  novice  teachers  is  a problem  for  school  districts.  Teacher  induction  programs  are  beneficial  in 
increasing  retention,  but  these  programs  take  time,  a commodity  in  short  supply  among  novice  teachers. 
Computer-based  support  has  been  suggested  as  an  alternative  resource.  This  paper  reports  the  findings  of  a 
survey  among  novice  teachers  in  the  rural  south-eastern  United  States  as  part  of  a needs  assessment  for 
developing  support  tools  and  programs  for  this  population. 

Introduction 

Teacher  turnover  is  an  unfortunate  problem  in  school  systems  across  the  United  States.  After  five 
years,  between  40  and  50  percent  of  all  beginning  teachers  leave  the  profession  (Ingersoll,  2003).  High  turnover 
rates  create  complex  problems  for  schools  including  less  stable  learning  environments  for  students  (DePaul, 
1998),  diverted  financial  resources  as  time  and  money  are  spent  in  recruiting,  hiring,  and  training  replacements 
(Berry,  Hopkins,  Thompson  and  Hoke,  2002;  DePaul,  1998),  and  limiting  districts  abilities  to  carry  out  long- 
term planning,  curriculum  revision  and  reform  (Halford,  1999).  Darling- Hammond  (2003),  in  examining  the 
issue  of  teacher  retention,  found  four  factors  influencing  teacher  turnover:  salaries,  working  conditions, 
preparation,  and  mentoring  support  in  the  early  years.  Mentoring  support,  in  particular,  has  been  shown  to 
reduce  attrition  rates  by  more  than  two-thirds  (NCTAF,  2003). 

One  method  of  mentoring  that  has  proven  beneficial  is  teacher  induction  programs.  Such  programs 
present  a structured  process  of  teacher  learning  in  the  first  few  years  the  teacher  is  in  the  classroom  (Berry, 
Hopkins,  Thompson  and  Hoke,  2002).  The  goal  is  to  assist  novice  teachers  in  developing  a wider  repertoire  of 
teaching  strategies  (Schafer,  Stringfield,  and  Wolffe,  1992),  stronger  classroom  management  skills  (Educational 
Resources  Information  Center,  1986),  and  strategies  for  dealing  with  behavior  and  discipline  problems  more 
effectively  (Moir  and  Bloom,  2003).  Components  of  successful  induction  programs  include:  professional 
development  (Feiman-Nemser,  2003;  Hinds,  2002;  Johnson  and  Kardos,  2002;  Wong,  2002);  interaction  with 
other  teachers  (Berry,  Hopkins,  Thompson  and  Hoke  2002;  Brewster  & Railsback,  2001;  Wong,  2002); 
principal/administrator  support  (Brewster  & Railsback,  2001;  Johnston  and  Kardos,  2002;  Wong,  2002);  new 
teacher  assessment  (Berry,  Hopkins,  Thompson  and  Hoke  2002;  Huling-Austin,  1992);  reduced  responsibilities 
(Berry,  Hopkins,  Thompson  and  Hoke  2002;  Renard,  2003;  Voke,  2002);  trained  mentor  support  (Berry, 
Hopkins,  Thompson  and  Hoke  2002;  Brewster  & Railsback,  2001;  Darling-Hammond,  2003);  and 
school/university  collaboration  (Berry,  Hopkins,  Thompson  and  Hoke  2002;  Brewster  & Railsback,  2001; 
Hinds,  2003).  In  addition  to  improving  retention,  induction  programs  have  been  shown  to  influence  teaching 
practices,  increase  teacher  satisfaction,  and  promote  strong  professional  development  and  collegial  relationships 
(Voke,  2002). 

Novice  teachers  have  constant  questions  and  concerns.  They  may  participate  in  scheduled  formal 
professional  development  sessions  as  part  of  their  induction  program;  however,  the  content  of  the  workshops 
may  not  be  of  immediate  use  to  the  teachers  and  may  not  answer  the  questions  that  plaque  new  teachers 
everyday.  Getting  answers  to  the  many  questions  they  have  is  also  impeded  by  the  novice  teachers’  concern  of 
how  their  colleagues  will  perceive  them  and  their  ability  to  teach  after  asking  the  questions  (Stapleton,  2002). 
Once  they  decide  to  ask  someone,  novice  teachers  often  find  that  they  lack  free  time  needed  to  ask  their 
question,  or  the  time  they  have  does  not  coincide  with  the  time  that  knowledgeable  teachers  are  available  to 
assist  them  (Stapleton,  2002). 

Computer-supported  interaction  may  be  an  appropriate  method  for  novice  teachers  to  obtain  the 
support  and  information  they  need  on  a daily  basis.  Listservs  and  email  (Ersinnan  and  Thornton,  1999)  as  well 
as  video  conferencing  (Thomson  and  Hawk,  1996)  have  been  used  to  provide  feedback  to  teachers.  Case 
libraries  have  also  been  developed  to  assist  in  learning  about  classroom  practice  (Jonassen,  Wang,  Strobel,  and 
Cernusca,  2003).  But  these  represent  isolated  tools.  The  integration  of  a number  of  tools  that  could  serve  as  a 
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virtual  community  for  novice  teachers  may  be  beneficial  in  providing  support  when  it  is  needed.  This  timely, 
multifaceted  support  may  help  to  increase  the  retention  rate. 

In  2002,  the  Golden  LEAF  Foundation  awarded  East  Carolina  University’s  College  of  Education  funds 
to  develop  a model  of  teacher  recruitment  and  retention  for  eastern  North  Carolina.  One  product  of  the  funding 
was  the  formation  of  the  Golden  LEAF  Educational  Consortium  (GLEC).  GLEC  is  a partnership  between  East 
Carolina  University,  University  of  North  Carolina  at  Pembroke,  Elizabeth  City  State  University,  Edgecombe 
Community  College,  and  eight  county  school  districts:  Bertie,  Edgecombe,  Greene,  Jones,  Halifax,  Perquimans, 
Robeson,  and  Washington.  All  of  these  counties  represent  rural  districts.  GLEC  was  charged  by  the  Golden 
LEAF  Foundation  to  develop,  implement,  and  evaluate  a model  of  teacher  recruitment  and  retention  and  to 
create  a toolbox  of  strategies  that  school  systems  could  use  to  recruit  and  retain  teachers.  One  strategy  proposed 
by  GLEC  was  the  development  of  a virtual  community  for  novice  teachers.  Prior  to  development,  a needs 
assessment  was  implemented  to  determine  if  novice  teachers  in  rural  areas  have  acceptable  access  to  the 
technology  needed,  and  to  determine  the  concerns  that  were  most  evident  to  teachers  during  the  first  years  of 
teaching.  This  study  reports  the  findings  of  a survey  among  novice  teachers  in  rural  north-eastern  North 
Carolina  as  part  of  a needs  assessment  for  developing  support  tools  and  programs  for  this  population. 

Method 


Participants 

Participants  included  novice  teachers  in  rural  north-eastern  North  Carolina  who  were  included  in  the 
Golden  LEAF  Educational  Consortium  (GLEC).  For  the  purposes  of  this  study,  novice  teachers  were  defined  as 
teachers  in  their  first  three  years  of  teaching;  however,  there  were  a small  number  of  participants  who  were  in 
their  first  year  of  teaching  in  the  county  but  not  their  first  three  years  of  teaching.  The  eight  participating 
counties  included:  Bertie  County  Public  Schools,  Edgecombe  County  Public  Schools,  Greene  County  Public 
Schools,  Jones  County  Public  Schools,  Halifax  County  Public  Schools,  Perquimans  County  Public  Schools, 
Washington  County  Public  Schools,  and  the  Public  Schools  of  Robeson  County. 

Instrument 

The  survey  instrument  contained  43  questions.  The  first  1 1 questions  were  demographic  questions 
dealing  with  age,  gender,  ethnicity,  teaching  and  educational  experience,  type  of  teaching  license  and  amount 
and  type  of  orientation  received.  The  next  12  questions  asked  about  the  support  the  new  teachers  received  from 
their  principals,  mentor  teachers,  and  New  Teacher  Coordinators  (ILT  Coordinators),  the  biggest  challenges 
they  had,  the  types  of  professional  development  they  received,  how  much  of  their  own  money  was  spent,  how 
much  time  they  spent,  and  if  they  were  planning  on  returning  to  teach  the  next  year.  The  next  five  questions 
asked  about  strategies  implemented  by  GLEC  and  classroom  management  challenges.  Ten  questions  were  asked 
to  ascertain  novice  teacher  access  to  and  use  of  computer  and  computer  related  resources.  The  last  three 
questions  asked  new  teachers  to  give  suggestions  for  improving  GLEC  strategies,  mentor  teachers  support  and 
new  teacher  induction. 

While  the  survey  questions  dealt  with  a number  of  issues,  only  those  questions  pertinent  to  the  technology  needs 
assessment  are  addressed  here.  Specifically,  this  article  focuses  on  novice  teachers’  answers  to  classroom 
challenges,  the  preparedness  of  the  teachers  to  handle  those  challenges,  their  sources  for  information  and 
advice,  their  satisfaction  with  the  answers  they  received,  and  their  access  to  technology  at  home  and  in  the 
school. 

Data  Collection  and  Analysis 

In  spring  2003,  surveys  were  sent  to  the  new  teacher  coordinators  (ILT  Coordinators)  in  the  eight 
participating  counties.  The  ILT  Coordinators  distributed  the  surveys  to  the  novice  teachers  during  a support 
meeting.  The  completed  surveys  were  then  returned  to  the  GLEC  Principal  Investigator  by  the  ILT  Coordinator. 
Of  the  370  surveys  distributed  to  novice  teachers,  225  returned  the  surveys  for  a response  rate  of  61%.  Surveys 
were  received  from  all  eight  counties  participating  in  the  program.  Surveys  were  coded  and  results  were 
analyzed  using  SPSS  (Statistical  Package  for  the  Social  Sciences). 
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Results 


Demographic  Results 

Forty-three  percent  of  the  respondents  were  age  21-25;  19%  were  26-30;  23%  were  30-39;  and  15% 
were  40  or  older.  Seventy-four  percent  of  respondents  were  female;  26%  were  male.  Fifty-six  percent  reported 
their  ethnicity  to  be  Caucasian;  30%  African-American;  9%  Native  Americans;  4%  Hispanic;  1%  did  not 
respond.  Ninety-five  percent  of  the  respondents  had  between  one  and  three  years  of  teaching  experience.  Fifty- 
three  percent  of  those  surveyed  entered  the  teaching  profession  through  an  alternative  licensure  program.  The 
respondents  were  divided  between  elementary  (45%),  middle  (31%),  and  secondary  (24%)  schools. 

Classroom  management  and  discipline  problems  rank  at  the  top  of  the  novice  teachers  concerns  at 
47.5%.  (see  Table  1).  Sspecific  examples  of  problems  included  talking  during  class  instruction,  keeping  the 
class  on  task,  teaching  and  disciplining  students  within  the  short  instructional  time,  following  through  on 
discipline,  dealing  with  disrespect  and  student  attitudes,  and  lacking  principal  support  on  discipline  decisions. 
Planning  and  teaching  to  the  state  standards  (15.5%),  meeting  the  needs  of  students  (13.6%),  and  school  policies 
and  procedures  (12.3%)  were  a distant  second,  third  and  fourth.  Other  challenges  listed  by  new  teachers 
included  lack  of  support/assistance,  time,  working  with  parents/staff,  paperwork,  amount  of 
requirements/responsibilities,  lack  of  resources,  obtaining  certification/licensure,  and  planning  and  teaching  for 
state  assessments. 

Table  1 Greatest  challenge  as  a new  teacher 

Percentage 


Classroom  management/discipline 

47.7 

Planning  and  teaching  to  the  state  standards 

15.5 

Meeting  needs  of  students 

13.6 

School  policy  procedures 

12.3 

Lack  of  support/assistance 

9.5 

Time 

8.6 

Working  with  parents/staff 

8.6 

Paperwork 

8.2 

Amount  of  requirements/responsibilities 

5.0 

Lack  of  resources 

5.0 

Obtaining  certification/license 

4.1 

Planning  and  teaching  for  the  state  assessments 

4.1 

The  follow-up  question,  "Did  you  feel  prepared  to  handle  these  challenges?"  was  also  asked  of  the  new 
teachers.  Fifty-three  percent  of  the  novice  teachers  answered  yes,  20.9%  answered  no  and  15.2%  answered 
"somewhat"/” sometimes".  Other  responses  to  this  question  indicated  that  after  receiving  support  from  their 
mentor  and/or  principal,  they  did  feel  prepared.  Others  mentioned  the  difficulty  they  had  at  the  beginning  and 
their  improvement  as  the  year  progressed.  One  new  teacher  mentioned  the  difficulty  in  applying  his/her 
knowledge  and  theory  into  a classroom  setting  while  still  remaining  positive.  At  lease  one  new  teacher  alluded 
to  the  fact  that  he/she  thought  he/she  was  prepared  “until  I realized  I was  not.” 

When  asked  where  the  new  teachers  went  to  get  answers  to  their  questions,  the  respondents  most  often 
turned  to  experienced  teachers  (91%)  (including  their  assigned  mentor  teacher  [87%])  and  administrators(72%) 
for  answers  to  their  concerns  but  they  also  used  print  resources  (43%),  teachers  at  other  schools  (39%),  family 
members  (32%),  fellow  novice  teachers  (30%),  friends  (29%),  online  resources  (27%),  and  former  classmates 
(20%)  and  professors  from  college  (17%).  (see  Table  2). 

Table  2 Resources  for  answers  for  the  new  teacher 

Percentage 


Experienced  teacher  at  my  school 

91.4 

Assigned  mentor  teacher 

87.3 

School  administrator 

72.4 

Print  resource  (book,  manual) 

42.5 

Teacher  at  another  school 

38.9 
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A family  member 

32.1 

A new  teacher  at  my  school 

29.9 

A friend 

29.4 

Online  resource 

26.7 

Someone  I went  to  college/university  with 

19.5 

A professor  from  my  college/university 

16.7 

Ninety-one  percent  of  respondents  were  satisfied  with  the  answer  they  received  when  consulting  those 
resources. 

Ninety-two  percent  of  respondents  indicated  that  they  had  daily  access  to  a computer  and  82.5% 
indicated  they  had  computers  in  their  home.  Ninety-two  percent  have  Internet  access  on  a daily  basis;  77.4% 
have  Internet  access  in  their  home  although  73. 1%  of  respondents  indicated  that  access  from  home  was  with  a 
dial-up  modem  connection. 

In  response  to  the  question,  "If  you  had  access  to  an  online  database  of  cases  with  solutions  based  on 
common  classroom  problems,  how  frequently  would  you  access  it?",  37.9%  indicated  they  would  access  it 
weekly  (see  Table  3). 

Table  3 Likelihood  of  accessing  an  online  database  of  cases 

Percentage 


Weekly 

37.9 

Occasionally 

28.0 

Monthly 

15.9 

Daily 

13.1 

Never 

5.1 

Discussion 

The  high  percentage  of  novice  teachers  who  listed  classroom  management  as  their  biggest  concern  in 
an  open-ended  question  indicates  how  heavily  it  affects  novice  teachers.  Classroom  management  concerns  and 
practices  should  be  integrated  more  into  pre -service  education  to  better  prepare  beginning  teachers  to  deal  with 
this  concern.  Notifying  principals  and  administrators  of  this  concern  may  assist  them  in  providing  guidance  and 
advice  for  novice  teachers.  Additionally,  mentor  teachers  need  training  so  they  are  aware  of  the  challenges 
faced  by  novice  teachers  and  have  strategies  to  help  these  novice  teachers  through  these  challenges.  It  is 
encouraging  that  many  of  the  overwhelmed  novice  teachers  finally  found  they  were  able  to  handle  their 
challenges  after  receiving  support  from  their  mentor  teacher  and/or  their  principal. 

It  is  also  interesting  that  when  reporting  resources  novice  teachers  used  to  answer  their  questions,  they 
listed  experienced  teachers  (91.4%)  in  their  school  over  their  mentor  teachers  (87.3%).  Possible  reasons  for  this 
difference  include  availability  of  the  experienced  teachers,  the  lack  of  concern  of  evaluation  by  the  experienced 
teacher,  and/or  new  teachers  seeking  out  the  experienced  teachers  they  feel  could  best  answer  their  questions. 

It  is  encouraging  that  91%  of  respondents  were  satisfied  with  the  answers  they  received  from  the 
various  sources  that  they  consulted.  These  teachers  are  primarily  turning  to  resources  within  their  schools 
(teachers  and  administrators)  but  they  are  also  not  restricting  their  search  to  that  venue.  Both  personal  and 
written  resources  play  a role  in  assisting  these  teachers  as  they  become  proficient  at  their  profession. 

Computer  access  appears  adequate  although  detailed  information  on  the  age  and  software  on  the 
computer  was  not  gathered  due  to  constraints  on  the  number  of  questions  that  could  be  asked  on  the  survey.  The 
dependence  on  dial-up  modems  for  Internet  access  indicates  that  high-bandwidth  media  such  as  video  and  audio 
may  need  to  be  kept  to  a minimum.  The  possibility  of  accessing  an  online  database  of  cases  dealing  with 
common  classroom  problems  was  welcome  by  the  group  with  only  5.1%  stating  that  they  would  never  access 
such  a system. 


Conclusion 

Given  the  high  rate  of  turnover  among  novice  teachers  all  avenues  for  increasing  retention  should  be 
explored.  This  study  indicates  that  beginning  teachers  have  many  questions,  particularly  about  classroom 
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management,  but  that  they  are  finding  the  answers  they  need  through  a variety  of  sources.  Novice  teachers  most 
frequently  turn  to  experienced  teachers  to  answer  their  questions.  For  this  reason,  it  may  be  helpful  to  used 
experienced  teachers  to  build  an  online  database  of  cases  dealing  with  common  classroom  problems.  In  this 
database,  experienced  teachers  could  talk  new  teachers  through  certain  situations  and  model  for  them  the 
thought  processes  they  used  in  deciding  how  to  react  to  the  situation.  Placing  this  information  online  would 
extend  the  opportunity  for  finding  answers  beyond  the  time  spent  in  the  school  building.  Novice  teachers  could 
find  the  answers  to  their  questions  when  it  is  convenient  for  them,  not  when  they  can  have  access  to  the 
experienced  teacher.  The  online  database  would  also  give  new  teachers  the  opportunity  to  get  advice  from 
several  teachers,  not  just  the  teacher  that  they  could  find  at  that  moment.  Building  a prototype  system  and 
testing  it  with  the  population  would  be  the  next  step  in  the  process. 
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Abstract 

Business  needs  in  multinational  corporations  call  for  courses  that  involve  problem  solving  and 
creating  and  sharing  new  knowledge  based  on  workplace  situations.  The  courses  also  need  to  be  engaging  for 
the  participants.  Blended  learning  at  Shell  International  Exploration  and  Production  involves  these  kinds  of 
outcomes  in  courses  designed  around  a workplace-learning  model.  Employees  use  a Web-based  system  to  make 
contributions  based  on  their  own  work  experiences  in  preparation  for  a face-to-face  session.  These 
contributions  then  feed  into  classroom  sessions  that  involve  collaborative  learning  where  the  workplace 
problems  and  experiences  of  the  participants  are  the  focus.  In  this  presentation  one  course  is  highlighted  that 
demonstrates  a number  of  game  -type  activities  based  on  the  participants'  own  workplace  experiences. 
Implications  for  other  courses  outside  of  the  Shell  context  are  discussed. 

Introduction  and  Problem  Statement 

In  the  workplaces  of  professionals  in  multinational  corporations  problems  and  challenges  continually 
arise  that  involve  creating  new  solutions  and  constructing  new  knowledge,  and  indirectly  involve  improving 
communication  and  understanding  among  colleagues  who  come  from  and  work  in  different  parts  of  the  world. 
Key  strategies  involve  capturing  and  sharing  both  the  explicit  and  tacit  knowledge  of  the  experienced  staff. 

Such  learning  can  be  either  formal  such  as  via  participation  in  structured  courses,  or  informal  such  as  via 
mentoring  and  coaching  in  the  workplace,  experiential  learning  (Brookfield,  1995)  and  participation  in 
"learning  communities"  (Wenger,  1998). 

Both  formal  and  informal  approaches  have  their  limitations  as  well  as  strengths.  Among  the  strengths 
of  structured  learning  are  guided  opportunities  to  learn  while  interacting  with  a new  set  of  peers,  broadening  the 
learner's  contacts  beyond  his  or  her  workplace  colleagues.  Among  the  strengths  of  informal  learning  in  the 
workplace  is  the  anchoring  of  learning  in  problems  and  situations  that  are  real  and  relevant  to  the  learner  and 
the  business  (Billett,  2001;  Collis  & Margaryan,  2003).  Collaborative  learning  within  courses  that  emphasize 
authentic  work-based  activities  can  integrate  both  these  sets  of  strengths  (Lim,  Tan,  & Klimas,  2001). 
Collaborative  learning  can  also  be  structured  to  involve  team-based  games  and  competitions,  as  methods  for 
engaging  and  motivating  the  learners.  In  this  paper  we  describe  a form  of  blended  course  emphasizing 
collaborative  learning  and  learner  engagement  applied  to  work -based  activities  and  illustrate  it  via  the  detail  of 
one  particular  course  for  technical  professionals  in  the  oil  industry. 

The  questions  addressed  in  this  paper  are: 

• Why  is  learning  involving  engaging  work-based  activities  an  appropriate  response  to  key 
needs  in  a multinational  organization? 

• What  does  such  a course  look  like  in  practice? 

• What  are  the  participants’  reactions  to  such  a course  and  the  implications  for  the  instructor? 

Learning  for  Experienced  Professionals  in  a Multinational  Corporation 

This  section  focuses  on  the  importance  for  corporate  learning  of  being  directly  relevant  to  business  and 
workplace  needs  and  at  the  same  time  being  engaging  and  motivating  to  the  participants. 

Learning  related  to  business  needsOrganizations  often  do  not  see  a direct  relationship  between 
business  results  and  their  investments  in  formal  corporate  learning  because  much  traditional  formal  learning  is 
limited  in  its  transfer  to  the  daily  workplace  (Smith,  2002).  Partly  this  is  because  formal  courses  too  often  focus 
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on  content  delivery  rather  than  on  building  upon  the  experiences  of  the  participants  for  peer  learning  and 
problem  solving  (Billett,  2001).  In  contrast  to  content  delivery,  corporate  learning  should  be  characterized  by 
knowledge  sharing,  capturing  experiences  from  the  participants,  reusing  them,  creating  new  knowledge,  and 
recognizing  and  solving  workplace  problems,  in  a process-oriented,  collaborative  manner  (for  a review,  see 
Collis  & Margaryan,  2003).  These  call  for  learning  processes  that  are  better  fostered  in  collaborative  learning 
situations  focusing  on  work-based  situations  rather  than  in  courses  characterized  by  content  delivery  (Seufert  & 
Seufert,  1999). 

When  supported  by  network  technology,  courses  based  on  collaborative  learning  centered  around  real 
workplace  problems  and  opportunities  can  begin  or  even  be  carried  out  in  entirety  when  the  participants  are  still 
in  their  workplaces.  Participants  can  be  directed  to  find  out  about  the  experiences  of  others  in  their  workplaces 
and  share  them  with  others  in  the  course  via  a course  Web  environment.  These  submissions  can  then  be  used  as 
the  basis  for  further  collaborative  activities  if  the  participants  come  together  in  a classroom  setting. 

Collaboration  occurs  in  a variety  of  ways:  among  the  course  participants  individually,  within  teams  of 
participants  with  similar  specialized  experiences,  or  between  teams  with  generically  similar  problems  but  from 
different  backgrounds.  Within  the  workplace,  collaboration  can  also  involve  the  participant’s  peers  and 
supervisor  and  relevant  others  in  the  corporation.  All  these  need  to  be  integrated  by  the  course  instructor  so  that 
sharing  and  learning  occurs.  This  approach  to  collaborative  learning  involves  the  use  of  Web-based  course 
environments  that  include  groupware  tools  such  as  shared  archives,  and  tools  for  structuring,  monitoring,  and 
motivating  overall  course  processes. 

Learning  as  engaging 

Although  professional  learners  acknowledge  the  importance  of  applying  learning  to  improve  their  own 
workplace  performance  and  also  to  eventually  have  a business  impact,  it  is  still  important  that  learners  feel 
personally  engaged  and  motivated  for  individual  learning  activities.  This  is  particularly  an  issue  when  part  or 
all  of  a course  takes  place  with  the  participants  staying  in  their  own  workplaces;  the  dynamics  of  a well- 
designed  classroom  session,  with  peer  interaction  and  the  stimulus  of  being  together  in  the  same  place,  are 
harder  to  achieve  when  participants  are  fitting  their  learning  activities  around  work  pressures.  Thus  it  is 
valuable  that  learning  activities  focused  on  the  workplace  occasionally  involve  elements  of  competition  or  team 
spirit  or  even  moments  of  fun.  Harris  (1991)  identifies  a number  of  ideas  for  motivating  learners  including:  (a) 
focus  on  teams  and  teamwork;  (b)  provide  rewards,  have  “winners”  and  near-winners;  (c)  have  learners 
participate,  be  actively  involved,  and  make  meaningful  personal  contributions  that  are  valued  by  others;  (d) 
stimulate  learners  to  find  and  defend  their  own  solutions  perhaps  within  a timed  competition  among  teams;  (e) 
make  learning  visual  and  tactile,  have  things  to  look  at  and  to  handle;  (f)  stimulate  tangible  thinking  as  well  as 
creativity;  and  (h)  maintain  a sense  of  energy  and  dynamism. 

In  the  literature,  there  are  a numberof  references  related  to  bringing  motivational  elements  into 
classroom  sessions  for  adult  professionals  through  game -type,  motivating  activities  and  also  other  discussions 
about  how  to  design  online  learning  to  be  motivating.  Smith  and  Drakeley  (2004)  for  example  assume  that 
Web-supported  learning  means  an  individual  learning  via  interacting  with  a computer,  and  thus  call  for  program 
designs  within  the  software  that  involve  elements  such  as  game -type  assessments,  animations,  and  real-life 
stories.  Sometimes  motivation  is  brought  in  through  the  way  a course  is  publicized  in  the  workplace.  Bailey 
(2003)  describes  a health-care  setting  in  which  employees  were  motivated  to  succeed  in  a mandatory  online 
course  through  promotional  strategies  in  the  workplace  such  as  posters  and  displays  in  public  areas  such  as 
cafeterias,  break  rooms,  meeting  rooms,  corridors  and  hallways;  and  by  providing  buttons  to  all  successful 
participants  and  making  the  wearing  of  the  buttons  a focus  of  in-house  communication  campaigns.  However, 
bringing  a motivating  environment  into  work -based  courses  that  partially  take  place  while  participants  are  still 
in  their  own  workplaces  is  not  yet  much  discussed.  In  particular,  designing  a course  so  that  Harris’  ideas  for 
motivating  learners  are  combined  with  the  need  to  deal  with  serious  problems  in  a professional  workplace  is  a 
challenge. 

To  integrate  these  ideas  of  making  learning  business  relevant  and  making  learning  engaging,  new 
course-design  principles  are  needed.  On  one  hand,  these  need  to  represent  good  design  for  any  professional 
learning.  Thus  fundamental  principles  of  good  learning  for  adult  professionals  should  apply.  Merrill  (2002)  has 
identified  five  “first  principles  of  instruction”,  which  fit  well  with  workplace-learning  situations.  He  argues  that 
“Learning  is  promoted  when: 

1.  Learners  are  engaged  in  solving  real-world  problems. 

2.  Existing  knowledge  is  activated  as  a foundation  for  new  knowledge. 

3.  New  knowledge  is  demonstrated  to  the  learner. 
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4.  New  knowledge  is  applied  by  the  learner. 

5.  New  knowledge  in  integrated  into  the  learner’s  world.”  (Merrill,  2002,  pp.  44-45) 

Merrill’s  principles  aim  clearly  at  business  relevance.  On  the  other  hand,  in  order  to  add  the 

engagement  dimension,  work -based  activities  related  to  real  workplace  problems  should  also  be  designed  to 
involve  aspects  such  as  those  identified  by  Harris  (1991):  team  competitions  with  rewards,  winners  and  near- 
winners; learner  participation,  active  involvement,  and  building  on  personal  contributions  from  the  participants. 
Participants  should  find  and  defend  their  own  solutions  perhaps  within  a timed  competition  among  teams. 
Tangible  thinking  as  well  as  creativity  should  be  stimulated  and  a sense  of  energy  and  dynamism  should  be 
maintained,  both  for  portions  of  the  course  carried  out  within  the  workplace  but  also  within  a classroom 
component.  An  example  of  how  this  is  being  done  in  practice  is  described  next. 

Work- based  Courses  at  Shell  International  Exploration  and  Production 

Professionals  in  multinational  organizations  gain  important  practical  experience  and  insights  over  time 
that  can  be  used  as  the  basis  of  engaging  and  business -relevant  learning  activities.  In  a large  organization  such 
as  Shell  International  Exploration  and  Production  (Shell  EP),  employees  with  the  same  general  job  title,  such  as 
Production  Technologist,  can  vary  considerably  in  their  practical  experience,  depending  on  the  part  of  the  world 
in  which  they  work.  After  five  to  ten  years  of  experience  in  the  company,  these  professionals  often  turn  to  the 
Shell  EP  Learning  & Leadership  Organization  (LLD)  for  a course  that  will  give  them  technical  re -energizing 
and  updating.  An  important  issue  for  Shell  EP  LLD  is  how  to  make  these  courses  meet  the  particular  needs  of 
the  individual  participants  including  their  need  for  engaging  learning  while  at  the  same  time  address  strategic 
business  goals  such  as  knowledge  sharing  and  building  on  experience.  Shell  EP  has  been  developing  and  using 
different  models  of  blended  learning  that  have  these  issues  as  focuses.  More  than  50  courses  have  been 
redesigned  and  more  than  80  distinct  course  events  have  run  within  Shell  EP  that  demonstrate  different  ways 
that  this  model  operates  in  practice  (Collis,  Margaryan,  & Cooke,  2004;  Collis  & Margaryan,  2003). 

Approximately  half  of  the  redesigned  course  events  involving  work-based  learning  supported  by 
technology  use  a blend  that  combines  a component  in  the  workplace  with  a classroom  component;  the  other  half 
use  only  the  workplace  component.  The  workplace  component  is  not  “e-learning”  but  rather  a series  of  work- 
based  activities  involving  collaboration  with  others  in  the  workplace  or  course  and  regular  submissions  into  the 
course  Web  environments.  The  submissions,  which  can  be  graded  if  appropriate  but  always  receive  at  least 
written  feedback  directly  into  the  Web  site,  can  be  used  as  discussion  points  and  resources  by  the  other 
participants.  Technology  is  thus  important  to  facilitate  this  approach.  A Web-based  course-management  system 
such  as  those  commonly  used  in  universities  is  a key  tool  (Collis,  2002).  This  system  combines  the  benefits  of  a 
learning-content  management  system  for  reuse  of  participant  submissions  with  tools  for  collaboration,  sharing, 
and  communication,  all  integrated  in  practice  under  the  leadership  of  an  instructor  who  him/herself  has  many 
years  of  experience  as  a technical  professional  in  the  company. 

The  Applied  Production  Technology  course 

The  course  "Applied  Production  Technology"  is  a course  for  Production  Technologists  in  the  oil 
industry  with  five  to  ten  years’  experience  in  the  company.  They  will  have  developed  different  types  of 
expertise  depending  on  the  situations  in  which  they  have  worked,  although  all  involve  skills  related  to  the 
technologies  that  are  used  to  bring  oil  from  where  it  is  found  to  production.  Production  Technologists  deal  with 
problems  involving  well  engineering,  the  planning  of  production  volumes  and  capacities,  the  design  of 
well/reservoir  interfaces,  sand  control,  artificial  lift  methods,  and  production-system  optimalization,  among 
others.  The  following  sections  describe  the  general  design  of  the  course  and  give  examples  of  the  engaging 
aspects  of  the  work-based  learning  activities  that  link  the  workplace  to  the  classroom  and  build  on  the 
experience  of  the  participants. 

Course  design 

The  course  begins  with  a three-week  period  of  interaction  and  submissions  via  the  Web-based  course 
environment  while  participants  are  still  in  the  workplace,  through  a variety  of  activities  in  which  participants 
identify  their  own  workplace  needs  as  well  as  their  own  experiences  that  will  be  relevant  to  others  in  the  course. 
This  is  followed  by  a two -week  classroom  session  in  which  frequent  team-based  activities  occur  based  on  the 
submissions  that  were  made  by  the  participants  when  they  were  still  in  the  workplace.  These  activities  are 
engaging  in  each  of  the  ways  that  Harris  (1991)  describes.  The  Web-based  system  continues  to  support  the 
interaction  and  reuse  of  submissions.  During  the  classroom  session,  differences  in  experience  among  the 
participants  are  capitalized  upon  in  game -type  activities. 
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When  participants  begin  the  course,  they  receive  the  URL  of  the  course  Web  environment  in  which  a 
large  number  of  course  resources  are  available  and  in  which  each  participant  has  access  to  his  or  her  own 
personal  biography  page.  All  course  resources  and  activities,  as  well  as  submissions  and  feedback  messages 
from  the  instructor,  are  integrated  into  a “Roster”  (see  Figure  1). 


Figure  1 . Portion  of  the  roster  of  the  Web  environment  supporting  the  production  technology  course 

In  Figure  1,  the  icons  to  the  right  of  matrix  cells  with  activity  instructions  indicate  links  to  all  the 
submissions  of  the  participants  as  well  as  the  feedback  given  to  those  submissions  by  the  instructor.  The 
instructor  can  determine  if  the  participants  can  see  each  others’  submissions  or  not. 

Via  a “News”  page  and  a “Course  Info”  page  as  well  as  the  “Roster”  the  participants  can  read 
information  about  the  course  and  a general  welcome  from  the  instructor.  An  excerpt: 

“ Think  of  the  course  as  a way  for  you  not  only  to  learn  but  to  demonstrate  that  you  are  a reed 
production  technologist.  The  contributions  that  you  make  in  the  course  will  help  others  ( perhaps 
through  the  magic  of  technology,  for  years  to  come).  For  this  new  version  of  the  course,  I want  to  get 
away  from  having  me  and  the  other  instructors  lecture  to  you  for  hours  on  end.  At  this  point  in  your 
career  as  a production  technologist  1 think  you  deserve  something  more  fun.  So,  I am  going  to  try  to 
keep  you  very  busy  and  working  hard.  I will  also  give  you  some  additional  incentives..  Your  “carrot” 
is  that  at  the  end  of  the  course,  I will  reward  those  people  that  have  done  well  on  the  course  with  a 
certificate  that  will  say  something  nice  about  their  accomplishments,  their  team  abilities  and  their 
attitudes  in  the  course.  Your  “stick”  is  that  the  course  will  have  several  competitive  team  events.  You 
don ’t  want  to  embarrass  your  team,  do  you  ? No,  I didn  ’t  think  so. . . Thus  here  is  your  checklist  of  pre- 
classroom activities  to  complete  before  you  arrive...  ” 

In  addition  to  five-ten  hours’  worth  of  reading  materials  and  one  activity  involving  an  orientation  to  the 
Web-based  environment,  14  work-based  activities  are  then  described,  each  requiring  a submission  from  the 
participant  into  the  Web  environment. 

Work -based  learning  activities 

Within  the  first  week  of  being  introduced  to  the  course  Web  site,  each  participant  is  shown  a list  of  the 
five  main  specialist  areas  involved  in  the  Applied  Production  Technology  course,  and  asked  to  indicate  to  the 
instructor  the  area  or  areas  in  which  he  feels  he  or  she  has  somewhat  specialized  knowledge  or  is  otherwise 
quite  experienced  and  knowledgeable,  but  also  the  area  in  which  he  or  she  most  needs  new  or  more  knowledge. 
The  instructor  uses  this  to  group  the  participants  into  different  sets  of  “specialists”  that  are  called  upon 
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throughout  the  course.  The  “specialists”  will  be  providing  questions  to  be  answered  by  experts,  short  stories 
about  their  own  experiences,  and  will  contribute  data  for  class  problems.  The  specialist  roles  come  up  in  many 
of  the  different  learning  activities  of  the  course.  Several  different  types  of  these  activities  are  described  next. 
Each  has  two  parts:  a portion  done  while  still  in  the  workplace,  and  a portion  done  in  a team  environment 
during  the  classroom  component  of  the  course. 

Completion-type  selection  challenge : In  production  technology,  completion  design  involves  the  steps 
and  decisions  needed  to  minimize  the  unit  technical  costs  of  a well.  It  involves  many  aspects  of  well  design. 
Some  course  participants  will  have  had  more  experience  than  others  at  completing  the  design  of  a well.  These 
“specialists”  for  this  activity  and  submit  via  the  course  Web  site  a problem  related  to  completion  design  that 
they  have  been  involved  with,  as  well  as  their  solution.  A Word  template  is  provided  to  ensure  that  the 
necessary  information  is  provided,  such  as  characteristics  of  the  rock  formation,  fluids,  and  reservoirs  involved. 
Then  in  the  classroom  session,  one  of  the  submitted  stories  is  chosen  and  the  “specialist”  sits  with  the  instructor 
to  describe  the  situation  to  the  other  participants.  The  other  participants  break  into  teams  and  have  a certain 
amount  of  time  to  construct  and  present  a solution.  The  specialist  and  the  instructor  score  the  results  and  give 
feedback.  This  is  repeated  for  each  of  two  additional  specialist  submissions  selected  by  the  instructor.  The  team 
with  the  highest  score  at  the  end  of  the  three  rounds  is  the  winner. 

Guidelines  for  the  activity  include:  Allow  questions  for  clarification  at  the  start  and  during  the  team 
work  but  brief  the  “specialist”  not  to  describe  his  solution  in  any  way.  The  “specialist”  who  contributed  the 
challenge  may  have  some  difficulty  deciding  which  team  has  the  best  reply  as  their  solutions  will  vary 
considerably  so  the  instructor  has  to  take  the  lead  in  the  scoring.  Score  each  presentation  relative  to  the  previous 
ones  so  that  a sense  of  excitement  builds.  Therefore  do  not  score  the  first  one  too  highly.  Allow  discussions  and 
comments  among  the  teams  as  each  challenge  is  presented.  Some  of  the  best  ideas  come  from  this. 

Production  Technologist  Quiz  Bowl.  Via  the  course  Web  environments,  participants  read  the 
following:  “How  much  PT  knowledge  do  you  have?  These  questions  are  an  example  of  those  that  I will  quiz  you 
with  during  the  PT  Quiz  Bowl  sessions  during  the  classroom  component  of  our  course.  They  are  derived  from 
materials  in  P264  (a  course  the  participants  had  already  taken ) and  this  course.  You  will  see  many  of  them 
again  when  1 ask  your  team  these  questions  during  our  classroom  sessions.  Test  yourself  now  and  see  how 
many  you  can  answer.  Use  this  also  to  determine  your  weak  areas  so  that  you  know  which  parts  of  the  course  to 
concentrate  on  most”.  Attached  to  this  is  a spreadsheet  with  about  60%  of  the  potential  questions.  The 
questions  are  general-knowledge  sorts  of  questions  with  well-defined  answers  (“Name  five  different  types  of 
sand  control”).  In  the  classroom  sessions,  the  instructor  creates  a feeling  of  a television  quiz  show,  building  up  a 
sense  of  excitement  about  the  Quiz  Bowl.  Beamed  on  the  wall  is  a large  title  (see  Figure  2).  The  instructor  sits 
on  a stool  to  the  right  of  the  title  screen  and  acts  as  the  quiz-show  presenter.  Each  of  the  teams  has  its  own 
session,  as  the  team  “in  the  hot  seat”  for  the  Quiz  Bowl. 


• PT  Quiz  BowU 


Figure  2.  Creating  the  atmosphere  for  the  Quiz  Bowl  sessions 

In  a rapid-fire  manner,  the  instructor/quizmaster  gives  a question  to  each  member  of  the  team.  If  that 
person  can  answer  within  a timed  count-down,  then  the  team  gets  two  points.  If  the  person  called  upon  cannot 
answer  but  someone  else  on  the  team  can,  then  the  team  scores  one  point.  The  scores  are  tallied  after  each 
question,  to  build  up  the  excitement.  There  are  20  questions  in  all,  so  a total  possible  score  of  40  per  team.  After 
each  team  has  had  its  session  (about  an  hour  each),  the  winning  team  is  rewarded.  The  purpose  is  to  encourage 
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a general-knowledge  review  in  a fun  atmosphere  in  which  those  with  specialist  knowledge  about  a question  can 
help  their  team  to  earn  at  least  one  point.  During  each  session,  someone  has  the  job  of  keeping  a “parking  lot” 
of  issues  needing  explanation  after  the  Quiz  Bowl  session  is  finished. 

Peer  assist  and  peer  review  challenges:  This  activity  is  based  on  the  premise  that  each  experienced 
person  wants  to  have  a chance  to  share  what  he  or  she  knows.  Previous  techniques  requiring  a presentation  from 
each  participant  took  too  much  time  (there  are  approximately  25  -30  participants  per  course  cycle)  and  there  was 
little  chance  for  discussion.  Prior  to  the  classroom  sessions,  each  participant  submits  a problem  that  is 
challenging  him  or  her  in  the  workplace.  The  instructor  gives  the  following  instructions  via  the  course  Web  site: 
“The  problem  you  bring  must  not  be  trivial  and  should  not  be  beyond  the  scope  of  your  role  in  the  business  to 
solve.  You  must  understand  the  problem  very  well  as  you  will  need  to  explain  it  well.  At  least  three  weeks  before 
the  classroom  session,  submit  your  Peer  Assist  problem  in  the  Web  environment,  with  enough  of  an  explanation 
that  I can  understand  the  problem  You  may  provide  as  much  detail  as  you  like  ( as  an  attachment  if  you  prefer). 
All  participants  will  be  able  to  see  everyone  else’s  Peer  Assist  problem.  ” During  the  classroom  sessions,  the 
teams  meet  together,  with  each  team  member  explaining  his  or  her  problem  and  the  others  giving  challenges 
and  support.  Brainstorming  techniques  are  emphasized:  The  instructions  as  given  in  the  Web  site  are:  “Each 
person  takes  five  minutes  to  write  down  any  brainstorm  ideas  for  solving  the  problem  on  sticky  notes.  Then  an 
open  session  starts  with  all  contributing  their  suggestions  and  working  on  the  solutions.  Keep  things  moving. 

Be  respectful.  No  ideas  are  bad  ones.  Maintain  an  environment  of  friendly  challenge  and  useful  support.  ” If 
multiple  viable  suggestions  are  found,  they  can  be  compared  by  placing  them  on  a matrix  grid  (see  Figure  3). 


High 

Potential 

Value: 

Low 


Difficulty-Risk  (Low  to  high) 


Figure  3.  Matrix  for  comparing  peer-assist  suggestions 

Each  team  posts  its  findings  in  the  Web  environment  so  that  they  can  be  referred  to  later  in  the  actual 
workplaces  of  the  participants.  Each  participant  submits  a reflection  to  the  course  Web  environment  that 
discusses  “(a)  a summary  of  the  help  you  received  from  your  peer  assist,  (b)  your  new  thoughts  on  the  way 
forward,  and  (c)  any  points  for  improving  the  process  for  your  next  peer  assist.  ” During  the  peer  assists, 
approximately  one  in  three  students  comes  up  with  a fresh  idea  or  an  actual  solution.  The  participants  also 
practice  key  principles  of  peer  assists  and  reviews:  the  value  of  proper  brainstorming  techniques,  of  giving  and 
accepting  peer  help,  and  the  value  of  external  perspective.  The  participants  are  very  pleased  with  this  activity. 

PT  Live!  : The  instructor  set  a goal  of  removing  all  the  lectures  from  the  classroom  sessions  because  in 
his  opinion,  “lectures  are  not  very  interactive;  for  experienced  staff  there  is  very  little  certainty  that  the  content 
is  at  the  right  level;  the  participants  are  not  at  all  responsible  for  success;  when  given  by  a variety  of  guest 
lecturers  the  quality  of  the  lectures  varies  considerably;  and  the  instructor  had  very  little  ability  to  influence  the 
guest  lectures ”.  Thus,  instead  of  giving  an  intensive  lecture,  each  guest  lecturer  makes  his  notes  available  ahead 
of  time  via  the  course  Web  environment.  The  instructor  invites  the  participants  who  are  serving  as  “specialists” 
for  the  topic  to  submit  questions  to  the  guest  lecturer  ahead  of  time,  again  via  the  course  Web  environment.  “In 
order  to  call  yourself  knowledgeable  about  a topic,  you  are  supposed  to  be  able  to  carry  on  a conversation  with 
an  expert  in  the  field.  Have  you  ever  seen  one  of  those  interview  shows  on  television?  Well  that’s  what  you  are 
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going  to  do  yourself.  You  are  going  to  interview  Mr.  X when  he  comes  to  one  of  our  classroom  sessions. 

Prepare  a set  of  questions  for  the  interview.  These  need  to  be  really  good,  interesting  questions.  They  need  to  be 
‘open  ' questions , meaning  the  answer  is  not  a one-word  answer,  [some  examples  are  given ].  You  will  need 
about  10-15  good  questions  for  the  topic.  Submit  these  via  the  course  Web  environment  at  least  three  weeks 
before  the  classroom  session.  We  are  going  to  provide  these  questions  to  the  inten’iewee.  ” The  classroom 
session  is  run  like  a television  talk  show,  where  the  instructor  serves  as  the  host.  The  guest  is  given  the 
opportunity  to  start  with  a 10-20  minute  overview  of  the  topic,  with  no  questions  during  this  time.  Then  the 
host  controls  the  entire  flow  of  the  session  just  as  a talk-show  host  would:  basing  the  interview  on  the  questions 
submitted  in  advance  by  the  “specialists”,  working  these  into  a logical  flow  and  balance.  There  are  short 
“commercial  breaks”  in  which  music  is  played  and  the  instructor/host  can  decide  whether  to  continue  with  a 
question  or  wrap  up  the  question  to  move  to  other  questions.  “Call-in  questions”  from  the  audience  can  also  be 
possible.  The  objective  is  to  have  the  participants  and  the  guest  experts  think  about  the  topics  ahead  of  time,  get 
out  of  the  experts  what  the  participants  want  to  know  via  the  opportunity  for  much  deeper  questions  than  would 
occur  during  a traditional  lecture,  keep  the  lecturing  short,  and  potentially  create  re  -usable  content. 

Creating  workplace  assignments  for  an  introductory  course : Production  technologists  in  Shell  EP 
first  take  a previous  course  (called  P264)  before  they  can  enrol  in  the  Applied  Production  Technology  course. 
The  purpose  of  the  Creating  Assignments  activity  is  to  re -study  the  topics  and  processes  in  P264  course,  and  in 
particular  the  new  set  of  workplace  activities  that  have  been  created  and  are  available  in  the  P264  Web 
environment,  and  then  do  the  following:  “ Have  a look  at  these  workplace  assignments  for  P264.  Do  they  look 
familiar?  1 hope  they  do  as  you  should  have  done  several  things  like  this  in  your  career  so  far.  And  these  relate 
directly  to  the  competence  profiles  of  production  technologists.  There  are  the  assignments  we  have  made  for  the 
new  P264  course.  Now  then.  Let's  help  out  the  PT  ‘newbies’  in  P264.  Pick  one  of  the  assignments  and  write: 

(a)  the  three  worst  mistakes  people  make  doing  this  type  of  task,  (b)  the  three  best  tips  you  can  think  of  to  help 
them  do  the  assignment , and  (c)  at  least  one  additional  thing  you  want  to  tell  them  about  doing  this  sort  of  task. 
Do  your  best  as  I intend  to  use  your  advice  on  the  P264  workplace  activities  as  content  for  that  course.”  This 
activity,  while  not  in  a game  spirit,  involves  creativity  and  engagement  through  being  able  to  use  one’s  own 
experience  as  input  that  will  be  studied  by  others.  In  addition,  the  assignments  for  the  P264  are  enriched  with 
experienced-based  insights,  beyond  what  the  instructor  may  have  had  time  to  add  to  the  write-ups  of  the 
activities. 

These  are  only  some  of  the  work-based  activities  in  the  Applied  Production  Technology  course.  In  all 
of  the  activities  certain  characteristics  are  present.  The  activities  allow  for  differentiation  in  the  ways  that 
participants  contribute  to  the  course.  Repeatedly,  their  different  backgrounds  and  experience  bases  are  tapped  in 
order  to  strengthen  the  process  of  learning  from  each  other’s  experience.  The  activities  done  before  the 
classroom  session  are  not  “self-study  e-modules”  but  instead  tools  by  which  participants  already  go  deeply  into 
the  course  topics  and  themselves  prepare  many  of  the  resources  that  will  be  built  upon  during  the  classroom 
sessions.  The  submissions  of  the  participants  are  not  only  reused  during  the  classroom  sessions  but  are  available 
via  the  database  underlying  the  course  Web  environment  for  reuse  as  examples  in  other  course  sessions  or  even 
in  other  courses  or  for  informal  coaching.  Participants  come  from  many  different  locations  worldwide, 
including  Nigeria,  Oman,  and  the  UK  and  USA  and  thus  bring  in  experiences  and  problems  that  vary 
geographically.  All  activities  focus  on  real  workplace  situations.  All  include  engaging  aspects  that  motivate  and 
stimulate  learning. 


Reactions  and  Implications 

The  approach  used  in  the  Applied  Production  Technology  course  is  highly  appreciated  by  the 
participants.  In  course  evaluations,  participants  consistently  give  the  course  and  the  instructor  high  ratings.  In 
an  evaluation  of  the  course  design,  using  Merrill’s  (2001)  first  principles  and  the  extensions  of  those  principles 
to  bring  in  strategic  goals  relating  to  engaging  work -based  activities,  the  course  ranks  very  highly.  Particular 
strengths  of  the  course  are  in  relation  to  Merrill’s  first  principle,  “Learners  are  engaged  in  solving  real-world 
problems”,  and  Merrill’s  fourth  principle,  “New  knowledge  is  applied  by  the  learner.”  In  terms  of  Harris’ 
guidelines  for  engaging  learners,  the  course  is  a strong  example  of  learner  participation,  active  involvement,  and 
building  on  personal  contributions  from  the  participants.  On  many  occasions  a game -type  or  competitive 
situation  is  involved,  between  teams.  Teamwork  is  stressed,  as  well  as  creative  answers  to  tangible  problems.  In 
terms  of  Web-site  design,  the  more -than  100  objects  in  the  course  Archive  are  well  organized  and  the  instructor 
makes  good  use  of  the  different  communication  possibilities  in  the  site  for  comments  to  the  entire  course  in  the 
News  and  to  individuals  via  feedback  messages.  The  course  environment  itself  is  reused  for  different  cycles  of 
participants,  with  only  minor  tailoring  when  needed. 
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The  basic  approaches  to  work-based  activities  and  engagement  are  not  specific  to  production 
technology.  They  can  be  applied  in  any  course  for  experienced  professionals,  particularly  when  those 
professionals  will  bring  with  them  many  different  sorts  of  experiences  within  a general  job  category.  The 
approach,  however,  requires  new  skills  of  the  instructor.  The  instructor  must  focus  more  on  activity  design  than 
on  content  presentation.  Also,  the  instructor  must  be  experienced  and  broadly  based  in  his  or  her  discipline  in 
order  to  respond  helpfully  and  critically  to  the  great  variety  of  workplace  experiences  that  will  be  reported  by 
the  participants.  The  instructor  needs  to  develop  a communication  tone  to  use  in  the  Web  site  that  sets  the 
atmosphere  for  team  work  as  well  as  individual  contributions.  And,  new  for  many  instructors  in  the  corporate 
context,  the  instructor  must  become  proficient  at  making  use  of  a Web-based  course  environment  system  as  the 
interface  among  all  aspects  of  the  course  during  the  portions  that  occur  when  participants  are  in  the  workplace. 
The  instructor  also  needs  to  develop  techniques  for  managing  his  or  her  own  time,  in  that  the  large  number  of 
submissions  coming  into  a course  Web  site  require  timely  feedback  but  at  the  same  time  cannot  be  allowed  to 
overwhelm  the  instructor.  The  instructor  will  also  need  to  interact  more  with  workplace  supervisors  of  the 
participants,  not  only  because  participants  will  need  time  and  space  to  carry  out  their  pre -classroom  activities 
but  also  because  many  of  these  activities  may  require  use  of  data  and  experiences  from  the  workplace  that  will 
have  to  be  cleared  by  the  supervisor  or  may  require  interpretation  by  others  with  experience.  All  of  this  requires 
new  approaches  to  course  design,  such  as  have  been  developed  at  Shell  EP  (Bianco,  Collis,  Cooke,  & 

Margaryan,  2002). 

Thus  learning  from  experience,  by  experienced  staff  involves  new  approaches  to  course  design,  new 
skills  of  the  instructor,  and  new  kinds  of  work  by  the  participants.  The  results,  in  the  opinions  of  those  involved 
in  Shell  EP  courses  such  as  Applied  Production  Technology,  are  well  worth  the  investment. 
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Determining  Whether  to  Migrate  to  Blackboard™ 
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Abstract 

In  1998,  a state  university  received  grant  funding  to  convert  their  Special  Education  Blindness  and 
Visual  Impairment  graduate  degree  program  to  an  online  format.  At  that  time,  commercial  web  course 
management  systems  were  not  accessible  to  blind  and  visually  impaired  users.  As  a result,  grant  designers 
developed  a custom,  accessible  platform,  which  led  to  accessibility  standards  for  online  courses  and  to  an 
award-winning  design  and  interface.  In  2002,  the  university  licensed  Blackboard™  and  encouraged  the 
migration  of  all  online  delivered  courses  to  this  standardized  system.  After  determining  that  the  newer  version 
met  accessibility  standards,  the  instructional  design  staff  conducted  a qualitative  return  on  investment  analysis 
to  evaluate  whether  the  migration  to  Blackboard™  would  cause  losses  in  instructional  and  interface  quality. 
This  paper  explores  the  pro  cess  for  developing  a qualitative  return  on  investment  and  how  the  benefits  and 
tradeoffs  were  analyzed  related  to  maintaining  an  internally  developed  system  versus  migrating  to 
Blackboard™. 


Introduction 

Traditional  methods  for  analyzing  whether  a decision  is  ultimately  a good  decision  have  focused  on 
measures  that  can  be  quantified  and  that  ultimately  contribute  to  a financial  bottom  line.  However,  in 
environments  that  may  not  be  driven  by  financial  bottom  lines  - educational  settings,  non-profit  organizations 
or  grant  activities  within  a higher  education  institution  - such  methods  for  analyzing  an  important  decision  fail 
to  capture  the  real  variables  in  the  decision.  Furthermore,  increasing  demand  for  attention  to  assessing  social 
impact  of  decisions  (Barbour,  1993;  Kaufman,  2000)  is  driving  the  need  for  newer  methods  that  take  into 
consideration  a broader  array  of  variables  and  the  ultimate  impact  of  a decision. 

The  ability  of  return  on  investment  (ROI)  and  cost-benefit  analysis  (CBA)  to  accurately  and  fully 
analyze  the  impact  of  a decision  is  being  called  into  question.  Barbour  (1993)  explains  that  ROIs,  CBAs  and 
risk  assessments  are  limited  because  they  often  leave  the  real  benefits  or  dangers  unassessed  since  those  are 
qualitative  aspects  of  a project  that  cannot  be  quantified.  Often  those  unassessed  benefits  or  dangers  are 
impacts  upon  human  lives  or  the  environment.  In  response,  agencies  such  as  the  Office  of  Technology 
Assessment  and  the  United  Nations  Development  Program  have  developed  mixed- method  analysis  procedures, 
such  as  the  “Human  Development  Index”  (Barbour,  1993,  p.  53),  that  analyze  both  the  quantitative  and 
qualitative  factors  of  decisions  or  policies.  In  business  and  industry,  Kaufman  (2000)  has  proposed  an 
Organizational  Elements  Model  as  a tool  companies  can  use  to  assess  their  ultimate  benefit  to  and  impact  upon 
society. 

While  every  decision  may  not  be  an  earth-shaking  one  requiring  analysis  of  societal  good,  there  are 
many  instances  where  qualitative  aspects  of  a project  must  be  assessed  and  analyzed  in  order  to  determine  the 
real  costs  and  benefits.  The  impact  of  a decision  upon  employee  attitudes,  public  perception  of  quality,  and 
even  changes  it  causes  in  processes  or  specific  design  standards  are  all  examples  of  more  qualitative  variables 
that  may  be  involved  in  a decision.  This  paper  explores  a specific  instance  where  a qualitative  ‘ROI’  process 
was  developed  in  order  to  assess  a decision  about  migrating  online  courses  from  one  platform  to  another.  While 
some  aspects  of  the  migration  issue  could  be  quantified,  many  could  not.  Still,  analysis  and  data  backing  the 
decision  were  needed  by  management.  We  will  describe  the  context  of  the  project  and  discuss  why  a qualitative 
ROI  was  appropriate.  We  will  also  define  ROI,  the  questions  we  investigated,  the  methodology  developed  to 
conduct  the  analysis,  and  the  findings  the  analysis  yielded. 

Qualitative  ROI  Project 

When  a regional  funding  organization  first  awarded  a state  university’s  Blindness  and  Visual 
Impairment  Program  grant  funding  in  1998  to  convert  its  Master’s  degree  program  to  an  online  format;  off-the- 
shelf,  commercial  web  course  management  systems  were  not  accessible  to  blind  and  visually  impaired  users. 
Because  10%  of  the  students  enrolled  in  such  a program  had  visual  impairments  and  one  faculty  member  was 
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blind,  it  was  imperative  that  the  grant  team  develop  a custom  web  course  management  system  and  identify 
online  synchronous  and  asynchronous  tools  that  were  accessible.  Over  the  length  of  three  years,  15  courses 
were  developed  on  this  custom,  internally -developed  platform,  and  a virtual  campus  web  interface  was 
developed  to  support  distance  students. 

Based  upon  the  success  of  the  program,  in  2001  the  university  was  awarded  a second  federal  grant, 
which  significantly  increased  the  scope  of  the  project.  It  provided  the  necessary  funding  to  continue  the  online 
program  in  Blindness  and  Visual  Impairment  and  to  convert  two  other  programs,  the  Deafness  and  Hard  of 
Hearing  and  Severe  Disabilities  Master’s  degrees  to  an  online  format.  This  federal  funding  was  also  used  to 
create  a national  center  related  to  disability  services  and  education.  The  online  Master’s  degrees  are  now  a part 
of  the  center’s  expanded  teacher  training  function.  Additionally,  the  center  contracts  with  other  universities  to 
support  the  conversion  of  their  low-incidence  disability  degree  programs  to  an  online  format.  Clearly,  the 
quality  of  the  online  courses  and  programs  - both  in  terms  of  instructional  design  and  accessibility  - formed  a 
cornerstone  of  the  center’s  work. 

At  the  same  time  the  center  received  this  federal  funding  and  expanded  its  efforts,  the  university  in 
which  the  center  is  housed  licensed  Blackboard™,  a commercial  web  course  management  system,  which  is 
maintained  and  administered  by  the  university’s  faculty  development  center.  During  the  fall  of  2001,  the 
center’s  staff  members  conducted  a research  study  to  determine  the  practical  accessibility  of  the  product.  The 
results  showed  that  the  majority  of  the  Blackboard™  interface  met  accessibility  standards  (Conn  & Ektermanis, 
2001). 

The  federal  funding  impacted  the  size  and  structure  of  the  instructional  design  team.  Three  additional 
instructional  designers  were  hired  to  support  the  expanded  missions  of  the  center.  One  challenge  was  to 
maximize  the  impact  of  the  new  instructional  design  team  members.  Even  with  an  increase  in  staff  members,  it 
was  difficult  to  address  the  issues  of  limited  faculty  control  that  were  an  inherent  part  of  the  internally- 
developed  system  and  the  increased  workload  of  maintaining  courses  in  the  Blind  and  Visual  Impairment 
program  as  well  as  the  extensive  work  needed  to  convert  the  Deafness  and  Hard  of  Hearing  and  Severe 
Disabilities  programs. 

The  new  instructional  design  staff  members  brought  varying  degrees  of  technical  expertise  thus 
making  it  necessary  to  consider  a migration  to  a commercial  web  course  management  system  with  a graphical 
user  interface.  Given  the  results  of  the  accessibility  research  study  and  the  changes  in  size  and  structure  of  the 
instructional  design  staff,  it  was  determined  to  be  an  appropriate  time  to  evaluate  the  benefits  and  tradeoffs 
related  to  maintaining  the  internally -developed  system  versus  migrating  to  Blackboard™.  Once  the  project  was 
determined  to  be  appropriate  and  necessary,  the  instructional  design  staff  conducted  a review  of  methods  to 
determine  an  appropriate  process  for  conducting  this  analysis. 

Literature  Review 

What  is  ROI? 

ROI  is  an  acronym  for  return  on  investment.  It  is  a method  for  measuring  the  worth  of  an  investment 
and  has  been  primarily  utilized  for  businesses  purposes.  In  the  1990s,  the  use  of  ROI  for  calculating  the  value 
of  training  and  performance  solutions  began  to  be  addressed  by  the  human  resource  development  and 
performance  improvement  fields  (Phillips,  1997).  In  a human  resource  development  context,  “ROI  practices 
are  a means  of  economically  connecting  the  performance  goals  of  efficiency  and  effectiveness  with  selected 
interventions  and  performance  results”  (Swanson,  1999).  The  literature  base  for  these  fields  advocates  using 
ROI  as  a means  of  measuring,  documenting,  and  communicating  the  value  of  support  interventions  to  both 
justify  projects  as  well  as  to  build  cases  for  continued  or  new  funding  (Pine  & Tingley,  1993;  Phillips,  1997; 
Stolovitch,  2002). 

Connecting  ROI  to  Kirkpatrick’s  Evaluation  Model 

ROI  has  been  connected  to  Kirkpatrick’s  (1998)  evaluation  model.  Kirkpatrick’s  original  model 
included  four  levels:  1)  Training  Reaction,  2)  Learning,  3)  Behavior,  and  4)  Business  Results.  Training 
Reaction  is  often  gathered  through  end  of  training  or  course  evaluations  and  captures  data  related  to  participant 
satisfaction  and  comments  related  to  how  the  training  or  education  may  transfer  to  work  situations.  Learning 
evaluation  data  attempts  to  capture  participant  perception  of  their  achievement  of  objectives  related  to 
knowledge,  skills,  and  attitudes.  Evaluations  of  behavior,  also  referred  to  as  Application,  investigate  changes  in 
work  performance.  Business  Results  evaluates  the  impact  of  the  interventions  on  related  business  variables. 

Phillips  and  Phillips  (2003)  added  two  new  levels  to  Kirkpatrick's  model  placing  “ROI”  and 
“Intangible”  at  the  fifth  and  sixth  levels,  respectively.  As  mentioned  earlier,  ROI  is  a process  for  measuring  the 


213 


costs  and  benefits  of  an  intervention.  Intangible  is  the  documenting  and  reporting  of  relevant  variables  that  are 
not  easily  converted  to  a monetary  value.  Although  the  levels  of  Business  Results  and  ROI  may  appear  similar, 
they  differ  in  that  Business  Results  attempts  to  measure  changes  in  the  business  related  to  the  intervention,  such 
as  productivity  or  attitudes  of  employees  or  profitability  levels.  ROI,  on  the  other  hand,  focuses  on  comparing 
these  identified  benefits  (or  disadvantages)  of  the  intervention  with  the  costs  of  implementing  the  intervention. 

Types  of  ROI  Evaluations 

There  are  several  different  types  of  ROI  evaluations  that  can  be  conducted.  Phillips,  Stone,  and 
Phillips  (2001)  directly  align  the  ROI  evaluation  options  with  Kirkpatrick’s  original  model,  which  forms  a 
framework  for  the  timing  of  data  collection  as  well  as  a consideration  of  the  levels  of  credibility,  accuracy,  cost 
to  implement,  and  difficulty  to  implement.  For  example,  when  conducting  an  ROI  measure  related  to 
Kirkpatrick’s  first  level  of  evaluation.  Reaction,  data  is  collected  during  and/or  at  the  end  of  the  training. 
Credibility  and  accuracy  of  these  measures  tend  to  be  lower  since  transfer  of  training  to  work  settings  has  not 
yet  occurred,  but  these  evaluation  measures  are  often  less  expensive  and  difficult  to  implement.  Collecting 
evaluation  data  related  to  Business  Results,  Kirkpatrick’s  fourth  level,  can  be  very  credible  and  accurate  since 
the  intervention  will  likely  be  implemented  by  this  point;  however,  these  types  of  measures  are  typically  more 
expensive  and  difficult  to  collect. 

In  addition  to  aligning  ROI  evaluations  with  Kirkpatrick’s  levels,  Phillips,  Stone,  and  Phillips  (2001) 
include  one  more  option  — Forecasted  ROI.  A Forecasted  ROI,  also  referred  to  as  worth  analysis  (Stolovitch, 
2002)  and  anticipated  ROI  (Parkman,  2002),  is  conducted  before  an  intervention  is  implemented  and  is  the  type 
of  ROI  employed  in  this  study.  It  can  help  provide  justification  for  a project  as  well  as  provide  baseline  data 
that  can  be  compared  with  post-project  results.  Forecasted  ROIs  are  based  on  estimations  and  therefore  may  be 
less  credible  or  accurate  than  ROIs  calculated  on  post-project  data.  However,  they  have  the  benefit  of  being 
inexpensive  to  develop  and  less  difficult  to  conduct. 

Sequence  and  Criteria  for  Conducting  ROI 

Before  conducting  a ROI,  a front-end  analysis  should  be  conducted  to  identify  “what  the  desired 
business  state  should  be,  what  the  current  or  actual  state  is  and  then  [to]  characterize  the  gap  between  the  two 
states  in  terms  of  magnitude,  value  and  urgency’’  (Stolovitch,  2002).  This  front-end  analysis  provides  data  for 
determining  an  appropriate  solution  and  clarifying  project  goals  in  terms  of  business  results.  The  results  of  the 
ROI  can  also  be  used  for  developing  project  evaluations  and  comparing  the  benefits  of  the  solutions  to  the 
potential  costs  (Parkman,  2002). 

Phillips  (1997)  describes  ten  criteria  for  conducting  a traditional  ROI.  These  criteria  form  a set  of 
guidelines  to  follow  when  investigating  the  ROI  of  a project,  product,  or  training.  These  guidelines  can  be 
summarized  as  1)  keep  the  process  simple  by  employing  practical,  feasible  methodologies;  2)  design  an 
economical  process  that  is  easy  to  implement,  has  the  potential  of  becoming  routine,  and  can  be  applied  to  a 
various  types  projects  as  well  as  to  both  pre -project  and  post-project  data;  3)  choose  evaluation  techniques  or 
research  methodologies  that  are  credible,  theoretically  sound,  and  based  on  accepted  practices;  and  4)  create  a 
process  that  can  utilize  all  types  of  data  and  include  the  costs  of  the  program. 

Qualitative  ROI 

Financial  factors  are  often  not  the  only  variables  that  need  to  be  considered  when  gathering  data  to 
estimate  or  judge  the  value  of  an  intervention.  Intangibles  are  variables  that  are  critical  to  the  overall  project  or 
solution  but  are  not  easily  converted  to  monetary  values.  As  mentioned  earlier,  Phillips  and  Phillips  (2003)  add 
‘Intangibles’  as  a sixth  level  to  Kirkpatrick’s  original  model.  Swanson  (1999)  states  “criteria  other  than  ROI  are 
being  used  to  gain  support  for  performance  improvement  programs.  Although  there  appears  a difference  of 
opinion  in  the  literature  regarding  whether  ‘Intangibles’  are  or  are  not  a ROI  measure,  we  chose  to  adapt 
procedures  to  create  a qualitative  ROI  process  given  the  mix  of  data  sources  available  and  the  context  of  our 
study. 

Swanson  (1999)  describes  several  qualitative  factors  that  should  be  considered. 

1.  Appropriateness  of  the  program  to  the  organizational  culture  and  tradition 

2.  Availability  of  the  program 

3.  Perceived  quality  of  the  program  design 

(p.  836) 

These  are  especially  important  to  take  into  consideration  when  working  with  non-profit  organizations  or 
educational  institutions  where  economics  may  not  be  the  key  driver  and  where  hard  program  costs  may  be 
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difficult  to  access  or  are  not  directly  financed  by  the  project’s  budget.  This  was  the  context  for  this  particular 
proposed  evaluation  project. 


Purpose  of  the  Qualitative  ROI  Project 

A front-end  analysis  had  been  conducted  to  verify  that  a course  management  system  was  (still)  needed 
to  deliver  the  three  low  incidence  disabilities  graduate  degree  programs  and  to  engage  in  contract  work  with 
other  universities.  Additionally,  the  version  of  Blackboard™  licensed  by  the  university  had  been  thoroughly 
tested  to  ensure  it  was  accessible  and  met  Section  508  standards.  Given  the  results  of  this  up  front  analysis,  the 
question  that  remained  was  what  would  be  the  benefits  and  tradeoffs  of  migrating  courses  to  Blackboard™ 
versus  continuing  to  use  the  internally  developed  web-based  course  management  system.  From  this  key 
research  question,  the  following  secondary  questions  were  developed.  Would  Blackboard™  (or  policies  related 
to  Blackboard™): 

1)  Decrease  course  development  time  for  the  instructional  design  staff,  so  that  more  time  could  be 
devoted  to  the  other  missions? 

2)  Allow  the  center  to  continue  to  deliver  high  quality  courses  with  cutting  edge  designs? 

3)  Allow  the  center  to  maintain  and  contribute  to  quality  instruction  in  the  areas  of  accessibility, 
increased  features,  and  increased  control  for  instructors? 

4)  Increase  the  center’s  return  on  monthly  fees  being  paid  to  the  university’s  information  technology 
and  faculty  development  departments? 

5)  Enhance  campus  relationships  between  the  faculty  development  department  and  the  center,  and 
add  value  to  the  university? 

6)  Support  the  center’s  ability  to  partner  with  other  institutions  for  delivering  courses? 

In  addition  to  exploring  the  questions  listed  above,  the  center’s  instructional  design  staff  also  felt  the  results  of 
this  study  could  prove  valuable  to  other  university  special  education  departments  with  whom  the  center 
consulted  and  who  were  considering  whether  to  develop  a course  management  system  internally  or  use  an  off- 
the-shelf  product. 

Methodology 

Going  into  the  project,  the  evaluation  team  realized  there  would  likely  be  many  intangible  variables 
and  many  other  variables  that  would  be  difficult  to  quantify,  given  the  fact  that  departments  within  institutions 
of  higher  education  typically  do  not  charge  internal  clients  for  the  services  they  provide  and  often  institutional- 
wide site  licenses  are  purchased  for  software.  Given  this  context  and  the  potential  variables  that  would  be  part 
of  the  overall  analysis,  a methodology  that  incorporated  the  ROI  criteria  and  qualitative  research  techniques  was 
employed.  Qualitative  inquiry  methods  are  an  appropriate  approach  for  descriptive  studies  and  for  researching 
practical  problems  (Creswell,  1998;  Merriam,  1988).  Given  the  descriptive  nature  of  qualitative  studies,  the 
findings  or  results  include  detailed  narratives  regarding  questions  being  researched.  These  descriptions  are 
intended  to  paint  a picture  for  the  reader  of  the  situation  or  entity  studied.  This  is  done  through  the  use  of  text 
and  images  as  well  as  through  quotes,  examples,  or  other  appropriate  artifacts  (Wilson,  1979). 

Project  team 

As  with  many  return  on  investment  projects,  a team  was  formed  to  conduct  this  study.  In  qualitative 
research  it  is  important  to  inform  readers  of  the  biases  the  researcher  or  researchers  bring  to  the  project. 
Informing  readers  of  researcher  bias  allow  them  to  draw  their  own  conclusions  regarding  the  trustworthiness  of 
the  findings.  The  research  team  for  this  project  consisted  of  two  instructional  design  center  staff  members  who 
were  also  pursuing  doctorate  degrees  in  Educational  Technology.  One  of  the  center’s  instructional  design  staff 
members  proposed  the  project  and  leaned  towards  migrating  to  Blackboard™.  The  second  center  instructional 
design  staff  member  had  been  with  the  project  since  the  receipt  of  the  first  grant  and  this  staff  member’s  work 
was  central  to  the  creation  of  the  internally  developed  course  management  system  interface;  she  was  hesitant 
towards  the  idea  of  migrating  to  Blackboard™.  In  addition,  the  project  team  included  five  other  instructional 
designers  with  varying  levels  of  expertise.  These  consultants  had  no  association  with  the  center. 

Data  collection 

Data  collection  for  qualitative  studies  often  involves  multiple  sources  (Creswell,  1998;  Merriam, 

1998).  These  sources  can  include  “documents,  archival  records,  interviews,  observation,  [or]  physical  artifacts” 
(Creswell,  1998,  p.  65).  Merriam  (1998)  states,  “interviewing  is  probably  the  most  common  form  of  data 
collection  in  qualitative  studies  in  education.  In  numerous  studies  it  is  the  only  source  of  data”  (p.  70).  Using 
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more  than  one  method  for  collecting  data  and  verifying  emerging  themes  reinforces  the  results  of  qualitative 
studies.  This  practice  of  collecting  and  analyzing  a variety  of  data  sources  is  called  data  triangulation  (Denzin, 
1978). 

The  data  collection  for  this  study  involved  multiple  sources.  We  interviewed  the  center’s  director  and 
conducted  two  separate  interviews  with  staff  members  from  the  faculty  development  department.  We  collected 
documentation  including  a slide  presentation  prepared  by  the  center’s  technology  manager,  which  explored 
technical  implications  of  using  various  combinations  of  servers  and  courseware  for  delivering  courses  online 
and  email  correspondence  with  the  Blackboard™  staff  member  responsible  for  accessibility  issues.  We  also 
accessed  two  websites,  the  Blackboard™  company  website  that  discusses  the  accessibility  of  the  tool  and  the 
Section  508  website,  a federal  government  site  dedicated  to  the  implementation  of  federal  legislation  for 
accessibility  of  multimedia  information.  Finally,  we  utilized  the  results  of  the  Conn  and  Ektermanis  (2001) 
study  that  had  been  conducted  to  investigate  the  practical  accessibility  of  the  Blackboard™  interface. 

Data  analysis 

The  interview  data  were  analyzed  using  qualitative  coding  techniques.  A characteristic  of  qualitative 
research,  as  defined  by  Merriam  (1988),  is  inductive  reasoning.  Inductive  reasoning  refers  to  the  emergence  of 
concepts  and  themes  through  data  analysis.  The  researcher  may  begin  the  data  analysis  with  an  outline  of 
possible  concepts  or  themes  he  or  she  expects  to  find,  but  these  initial  codes  are  often  revised,  eliminated,  or 
added  to  through  the  coding  process. 

The  detailed  interview  notes  were  transcribed  and  were  read  and  re-read  by  the  project  team.  The 
transcriptions  were  then  analyzed  using  basic  qualitative  analysis  methods.  The  basic  qualitative  analysis 
method  followed  by  the  team  included  first-level  coding,  which  involves  how  one  differentiates  and  combines 
the  data  the  researcher  has  retrieved  and  the  reflections  one  makes  about  the  information.  These  codes  are 
designed  to  be  descriptive  labels  for  identifying  chunks  of  information  (Miles  & Huberman,  1994).  The 
research  questions  were  used  to  guide  the  analysis  of  the  data,  and  were  the  basis  for  the  initial  list  of  codes 
used  to  analyze  the  interview  transcriptions.  A content  analysis  of  the  documentation  collected  was  also 
conducted  using  a first-level  coding  method  to  triangulate  the  data  collected  through  the  interviews. 

For  first- level  coding,  the  team  wrote  all  the  data  on  large  pieces  of  paper  that  were  then  posted  on  the 
walls  of  the  room.  One  member  of  the  team  typed  these  pieces  into  an  electronic  format  as  the  group  worked. 
Once  all  the  data  was  posted  on  the  wall,  the  group  started  by  simply  numbering  each  piece  of  data.  The  first 
data  piece  listed  received  number  one.  If  the  next  piece  was  similar  to  something  already  numbered,  it  received 
that  same  number.  Otherwise,  a new  number  was  introduced.  In  the  first  round  of  coding,  six  categories  of  data 
emerged.  The  person  entering  it  electronically  reorganized  the  pieces  into  those  six  categories. 

The  next  step  in  the  analysis  process  was  second-level  pattern  coding.  Second-level  pattern  coding  is  a 
method  for  grouping  first-level  codes.  Pattern  coding  is  used  to  identify  emergent  themes  or  explanations.  The 
primary  purpose  of  second-level  pattern  coding  is  to  assist  in  getting  to  the  next  level  of  analysis  - beyond 
simple  description  (Miles  & Huberman,  1994). 

For  second-level  coding,  the  team  printed  out  the  initial  six  categories  and  placed  these  on  the  wall. 
They  then  discussed  what  the  categories  would  be  called  and  whether  categories  should  be  maintained  or 
whether  some  overlap  still  existed  between  categories.  After  further  analysis,  the  project  team  reduced  the 
categories  to  three  core  themes:  Quality,  Time  and  Cost.  Once  the  first-level  coding  and  second-level  pattern 
coding  was  complete,  the  project  team  analyzed  the  results  and  synthesized  data  into  a descriptive  report 
delivered  to  the  center’s  management  team  that  included  quotes,  examples,  and  images  to  convey  the  findings 
and  recommendations  of  the  study. 


Findings 

Based  on  our  analysis,  the  issues,  concerns  and  solutions  collected  from  all  sources  were  grouped  into 
three  key  themes:  Quality,  Time  and  Cost.  Figure  1 visually  depicts  the  relationship  that  emerged  between 
these  three  themes.  The  findings  of  this  study  indicate  that  the  main  considerations  short-term  were  the  issues 
of  the  loss  of  the  center’s  identity  and  ‘sense  of  place,’  and  the  impact  on  instructional  designers’  time.  The 
findings  also  pointed  to  concerns  related  to  long-term  sustainability  of  the  online  courses  as  well  as  different 
roles  and  time  investments  for  the  center’s  staff.  In  addition,  benefits  of  the  migration  to  Blackboard™ 
emerged  as  well  as  several  specific  issues  that  were  documented  as  Recommendations.  Table  1 contains  a 
summary  of  the  findings  categorized  as  ‘costs’  and  ‘benefits.’ 


216 


Figure  7. Visual  representation  of  the  relationship  between  the  three  major  themes  emerging  from 


Table  1.  Summary  of  Findings  Categorized  as  Costs  and  Benefits 


[Variable 

Costs 

Benefit 

Quality 

Accessibility  of 
Blackboard™ 

No  longer  an  issue  except  for 
minor  problems  with  the  interface 
and  the  chat  room  tool 

With  increased  collaboration 
between  the  center  staff  and  the 
faculty  development  department 
the  center’s  accessible  chat  room 
tool  could  be  made  available  to  the 
entire  university 

Intetface  Design 

Would  lose  ‘sense  of  place’  and 
community  designs 

Instructor  control 

Instructors  would  have  more  direct 
control  over  making  changes  in 
their  courses;  a feature  that  was 
not  available  in  the  internally 
developed  system 

Instructional 
Design  Quality 

Loss  of  infrastructure  that 
supported  webs  of  information  and 
data  pieces 

Technical  Issues 

University  server  less  stable,  more 
down  time 

More  portability  across  programs 
and  universities 

Development 

Support 

Long-term,  university  funded 
support  for  course  development 
and  maintenance 

Time 

Course 

Development 

Decrease  course  development  time 
for  the  center’s  staff  since  the 
faculty  development  department 
could  assist  with  course 
development 

Collaboration 

The  center’s  staff  would  likely 
need  to  spend  more  time 
participating  in  faculty  and  staff 
development  trainings  to  address 
issues  of  accessibility 

This  would  allow  the  center  to  add 
value  to  the  university  community 
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Help  Desk 
Support 

Student  confusion  surrounding  the 
accurate  logon  and  password  for  a 
specific  course 

With  the  migration  to  one  system 
for  the  university,  the  students 
would  more  likely  become 
accustomed  to  the  appropriate 
logon  and  password;  additionally, 
the  university  help  desk  would  be 
able  to  assist  all  students,  giving 
students  one  central  place  to 
contact  or  for  instructors  to  refer 
students  to 

Cost 

Multimedia 

Development 

Faculty  development  department 
provides  (free)  audio,  video  and 
graphic  development  services 

Information 
Technology  Fees 

The  center  was  required  to  pay 
$40  per  credit  hour  to  the 
university’s  information 
technology  department  for  support 
services,  but  the  center  was  not 
receiving  any  value  for  these  fees 
since  all  course  development  and 
support  to  students  and  instructors 
was  provided  internally 

Risk  Factors 

Unknown  outcome  of  annual 
contract  renegotiations  between 
the  university  and  Blackboard™ 
which  could  potentially  require 
distribution  of  Blackboard™ 
product  fees  to  the  department 

Quality 

The  migration  of  courses  to  the  university’s  web  course  management  system.  Blackboard™,  raised 
several  concerns  related  to  the  quality  of  current  courses,  specifically  issues  of  handicapped  accessibility  and 
good  instructional  and  visual  design.  Based  on  the  data  analysis,  it  was  determined  that  these  concerns  could  be 
addressed  through  proposed  solutions  or  balanced  by  gains  from  the  proposed  migration. 

Accessibility:  Earlier  accessibility  issues  related  to  the  Blackboard™  interface  had  been  addressed,  and 
most  course  components  now  met  federal  accessibility  guidelines  under  Section  508  which  stipulates  that 
electronic  and  information  technology  should  be  programmed  in  such  a way  that  individuals  with  disabilities 
can  access  and  use  the  information  and  data  in  a way  that  is  comparable  to  the  access  and  use  by  individuals 
without  disabilities.  Some  features  of  the  Blackboard™  interface  were  still  not  accessible,  though,  such  as  the 
chat  rooms.  Interviews  conducted  for  this  ROI  revealed  that  university  staff  were  willing  to  allow  the  center  to 
link  in  its  own  custom,  accessible  chat  rooms  and  other  tools  and  even  make  those  tools  available  to  users 
across  campus,  adding  further  benefit  to  the  entire  campus. 

Interface  Design:  The  custom,  internally-developed  interface  featured  an  identity  and  ‘sense  of  place’ 
that  was  designed  to  be  extremely  user  friendly.  The  original  interface  created  a sense  of  community  by 
developing  a virtual  campus  around  the  online  courses  and  programs,  and  an  infrastructure  that  allowed  students 
to  connect  with  each  other  outside  of  class  or  with  outside  experts  for  informal  discussions,  much  like  a 
physical  university  center  would  host  social  and  informal  events.  Students  could  also  access  ‘offices’  and 
‘buildings’  that  they  needed  to  be  successful  in  their  studies,  such  as  financial  aid,  the  library  and  faculty 
offices.  This  instructional  strategy  of  community  was  supported  with  visuals  and  identifiers  that  all  created  a 
‘sense  of  place’  where  the  students  felt  like  they  were  a part  of  a program  and  a university,  not  just  taking  online 
courses.  Within  classes,  a visual  interface  resembling  a classroom  had  been  developed  that  included  pictures  of 
faculty  and  other  features  that  helped  students  adapt  more  quickly  to  this  new  innovation  of  online  learning.  All 
this  work  had  been  created  based  on  research  on  learning  communities  (Wenger,  1998;  Palloff  & Pratt,  1999) 
and  change  facilitation  (Rogers,  1995;  Hall  & Hord,  2000). 

A concern  that  surfaced  during  this  study  was  that  this  distinctiveness  would  likely  be  lost  with  the 
move  to  Blackboard™.  However,  benefits  to  be  gained  with  the  migration  appeared  to  offset  this  concern.  For 
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example.  Blackboard™  would  allow  faculty  more  control  over  course  changes  and  updates  as  well  as  a wider 
array  of  course  tools  and  features.  According  to  the  literature  base  on  change,  such  a form  of  empowerment 
would  be  critical  for  faculty  adoption  of  the  innovation  because  it  fosters  buy-in  and  improves  success 
(Ellsworth,  1995)  and  allows  the  stakeholders  to  participate  in  the  very  technology  that  will  impact  their  work 
(Ellsworth,  1997;  Ely,  1990).  A final  benefit  of  offsetting  the  concern  over  loss  of  distinctiveness  was  that  by 
using  the  university’s  official  contracted  system,  long-term  support  for  course  development  and  maintenance 
was  assured,  if  funding  would  not  be  available  to  support  center  instructional  design  staff  in  the  future. 

Instructional  Design:  In  addition  to  accessibility  and  interface  design  concerns,  were  concerns  of 
instructional  design  quality.  Courses  had  been  redesigned  for  online  learning  based  on  cognitive  apprenticeship 
and  situated  cognition  principles  (Brown,  Collins,  & Duguid,  1989;  Herrington  & Oliver,  2000).  Instructional 
designers  had  built  scavenger  hunts  using  the  server  technology,  created  information  databases  students 
collaboratively  populated  as  course  projects,  and  even  built  a set  of  scaffolded  case  studies  and  support  tools 
that  all  made  extensive  use  of  the  custom,  internally -developed  platform.  In  addition  to  the  concern  as  to 
whether  these  quality  learning  experiences  could  be  preserved  using  the  Blackboard  platform,  was  the  practical 
consideration  that  migration  of  this  custom  content  would  not  be  straightforward,  but  would  require  time  and 
expertise  to  ensure  a well-managed  process.  However,  as  with  the  accessibility  issues,  the  qualitative  ROI 
revealed  that  the  center  could  maintain  complex  structures  on  its  own  server  and  link  to  those  from  within  the 
courses  on  the  Blackboard™  server.  Thus,  the  center  could  capitalize  on  the  benefits  of  Blackboard™,  but 
maintain  the  past  work  and  future  flexibility  that  had  contributed  to  its  reputation  for  quality. 

Technical  issues:  Additional  quality  concerns  centered  on  technical  issues.  The  university’s 
information  technology  server  was  viewed  as  less  stable  than  the  server  running  the  custom,  internally- 
developed  course  management  system.  The  university’s  information  technology  server  had  been  down  for  two 
months  total  out  of  the  year  when  the  center’s  server  was  not,  one  of  those  downtimes  occurring  during  the 
critical  time  when  courses  started.  Furthermore,  the  listserv  functions  within  Blackboard™  or  from  information 
technology  were  either  not  as  reliable  or  did  not  provide  the  same  capabilities  as  the  original  server  structure. 
However,  the  center  would  be  able  to  institute  its  own  policy  and  control  related  to  backing  up  courses. 

Portability:  One  final  advantage  of  migrating  the  center’s  online  courses  to  Blackboard™  included 
cross -institutional  portability.  By  using  a common  system,  other  universities’  special  education  departments 
could  partner  with  the  center  to  develop  courses.  While  a custom  solution  provided  the  center  a high  degree  of 
strength  and  flexibility,  other  programs  did  not  have  the  resources  to  maintain  something  similar.  Their  need  to 
be  on  a standardized,  university-maintained  system  outweighed  the  advantage  of  the  internally  developed, 
custom  solution. 

Time 

One  of  the  major  issues  we  investigated  was  whether  switching  to  Blackboard™  would  decrease 
course  development  time  for  the  center’s  staff.  Through  this  process,  we  discovered  that  one  of  the  services  the 
faculty  development  department  offers  (free  of  charge)  is  to  do  course  development  for  faculty  members.  To 
date,  mo  st  faculty  members  had  opted  to  develop  their  own  courses,  so  this  service  was  not  being  widely  used. 
Other  issues  that  surfaced  related  to  time  were  the  requests  by  the  faculty  development  department  for  help  with 
training  their  staff  on  development  techniques  related  to  accessibility  as  well  as  assisting  with  their  faculty 
development  courses  on  issues  related  to  accessibility. 

Based  on  our  analysis,  it  appeared  that  in  the  short-term  switching  to  Blackboard™  might  increase 
work  for  the  center’s  staff  to  meet  the  requests  made  by  the  faculty  development  department.  Long-term,  the 
data  appeared  to  support  the  objective  of  decreasing  course  development  time  for  the  center’s  staff.  Although 
these  results  did  not  support  decreased  staff  workload  in  the  short-term,  the  migration  would  likely  enhance 
campus  relationships  between  the  faculty  development  department  and  the  center,  and  add  value  to  the 
university. 

The  final  issue  related  to  time  that  surfaced  was  the  logon/password  confusion  encountered  by  students 
at  the  beginning  of  each  semester.  This  was  an  issue  that  the  faculty  development  department,  the  center,  and 
Special  Education  instructors  currently  dealt  with  each  semester  regardless  of  the  web  course  management 
system.  The  center’s  internally  developed  system  required  unique  course  passwords  that  were  reset  each  time 
the  course  was  taught.  By  migrating  to  Blackboard™,  students  would  be  able  to  use  their  university  logon  and 
password.  Additionally,  the  university  help  desk  would  be  able  to  respond  to  requests  for  help  regarding  the 
logon  and  password,  providing  another  source  of  help  for  students  beyond  the  center’s  staff  members  and  the 
course  instructor. 
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Cost 


Cost  is  an  inevitable  issue  when  considering  the  use  of  new  tools  and  services.  Upon  analysis,  it  was 
found  that  the  data  pertaining  to  cost  could  be  organized  into  three  categories  or  cost  issues  for  this  evaluation 
project,  Free  Services,  Return  on  Information  Technology  Fees,  and  Ris  k.  The  data  regarding  Free  Services 
revealed  the  faculty  development  department’s  willingness  to  offer  free  audio,  video  and  graphic  development 
to  supplement  the  instruction.  These  services  would  sustain  the  center’s  goal  for  providing  students  with 
authentic  and  innovative  learning  environments  through  the  utilization  of  technologically  advanced  tools. 

The  factors  fitting  under  the  Return  on  Information  Technology  Fees  exposed  a current  lack  of  return 
for  fees  paid  by  the  center  to  the  university’s  information  technology  department.  The  center  was  required  to 
allot  $40  per  credit  hour  to  the  university’s  information  technology  department  for  support  services.  At  the  time 
this  qualitative  return  on  investment  analysis  was  conducted,  the  center  had  not  been  able  to  offer  the 
university’s  information  technology  department  support  to  students  and  faculty  since  the  center  staff  maintained 
the  custom  course  management  system.  Migration  to  Blackboard™  would  provide  the  opportunity  for  the 
center  to  take  advantage  of  the  support  services  offered  by  the  information  technology  department,  thus  creating 
a better  return  on  investment  for  the  fees  paid  by  the  center. 

Based  on  the  results  of  the  analysis  of  cost  data,  the  outcome  appeared  to  be  in  favor  of  a migration  to 
Blackboard™,  with  the  exception  of  the  Risk  factors.  Risk  factors  associated  with  cost  related  to  the  unknown 
future  outcomes  of  contract  renegotiations  between  Blackboard™  and  the  university.  Renegotiations  occurred 
annually,  and  terms  of  a new  contract  could  at  some  point  include  a dispersal  of  cost  onto  the  departments  and 
organizations  that  use  Blackboard™. 

Recommendations 

Throughout  the  data  analysis  process,  ‘action  items’  continued  to  emerge  that  would  need  to  be 
addressed  should  the  center  decide  to  migrate  to  Blackboard™  for  online  course  delivery.  The  evaluation 
project  team  brainstormed  solutions  that  could  be  implemented  to  address  the  ‘costs’  (e.g.,  concerns  or 
disadvantages)  highlighted  in  the  findings.  Presented  in  Table  2 is  a summaiy  of  the  issues  that  emerged  from 
the  data  as  well  as  recommended  solutions. 

Table  2.  Summaiy  of  Issues  Related  to  Migrating  to  Blackboard ™ and  Recommended  Solutions 


Issue 

Recommended  Solution 

Loss  of  identity 

■ Providing  a link  from  the  main  Blackboard™page  to  the  center  information, 
stating  ‘supported  by  the  center’  in  some  fashion  - (create  a general  page  for  all 
courses) 

■ Put  the  center’s  ‘design  element’  on  Blackboard™  page 

■ Support  accessibility  through  training  faculty  development  department  staff  and 
instructors 

No  sense  of 
community 

■ With  every  center  course,  create  a link  for  all  students  to  access  the  center’s 
community  (create  a general  page  for  all  courses) 

■ Encourage  faculty  to  develop  community  through  course  design 

■ Use  orientation  for  students  at  beginning  of  course  to  inform  them  of  the  center’s 
virtual  campus 

Managing  the 
load 

■ Migration  Plan 

Incorporate  change  management  into  the  plan 
=>  Limit  the  number  of  transitions  and  course  management  interfaces  for 
students  and  instructors 

Policies 

■ Negotiate  policies  related  to: 

=>  Courses  being  taught  for  other  universities 

Continuing  support  for  online  courses  and  migration  to  any  new  system(s)  if 
the  university  drops  the  contract  with  Blackboard™ 

Relationships 

■ Establish  and  foster  clear,  positive  communication  lines 
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■ Support  accessibility  through  trainings  sponsored  by  the  faculty  development 
department  and  through  individual  consultation  with  instructors 

■ Extend  accessibility  support  to  library  and  web  manager  (for  entire  campus) 

Cutting  edge 
design 

■ Not  compromised  by  Blackboard™  if  Educational  Technology  staff  mediates 
between  faculty  and  Blackboard™ 

Stability  of 
servers/  longevity 
of  Blackboard™ 
contract 

■ Establish  systematic  back-up  procedure  for  all  courses 

Summary  of  Findings 

Thus  the  findings  showed  that  the  migration  to  and  use  of  Blackboard™  would  not  jeopardize  quality 
of  online  courses  and  would  likely  increase  quality  given  the  additional  features  of  the  system.  Over  time,  use 
of  Blackboard™  would  translate  into  more  time  for  the  center’s  Educational  Technology  staff  members  to 
devote  to  broader  instructional  design  efforts  rather  than  course  development  issues.  Under  the  university 
model  and  contract  in  place  at  the  time,  the  migration  to  and  use  of  Blackboard™  would  incur  no  cost  to  the 
center  and  would  actually  increase  return  on  monies  already  being  paid  to  the  faculty  development  department. 
Additionally,  if  sources  of  funding  for  Educational  Technology  staff  were  decreased  or  cut  completely,  the  use 
of  Blackboard™  would  ensure  longevity  of  courses  beyond  any  particular  grant.  Major  and  minor  issues 
related  to  Blackboard™  could  be  addressed  through  policies,  procedures,  or  products  as  identified  in  the 
Recommendations  section.  Therefore,  it  was  the  recommendation  of  the  project  team  that  the  center  migrate  its 
courses  to  the  Blackboard™  system  and  implement  the  recommended  solutions  to  make  such  migration  as 
smooth  and  beneficial  as  possible. 


Conclusion 

Overall,  the  qualitative  approach  aligned  well  with  the  type  of  data  the  team  was  able  to  collect  for  this 
evaluation  project.  We  were  able  to  address  the  intangible  issues  related  to  delivering  quality  courses.  Only  a 
few  hard  financial  costs  were  gathered  as  part  of  the  data  for  this  study;  however,  this  qualitative  approach 
could  be  easily  combined  with  more  traditional  financial  ROI  calculations  if  it  was  appropriate  for  the  project 
and  the  financial  data  was  available  to  analyze.  The  final  report  did  provide  the  center’s  management  team  with 
detailed  analysis  of  both  the  benefits  and  disadvantages,  allowing  them  to  make  a data-based  decision  regarding 
whether  or  not  to  migrate  to  Blackboard™  or  remain  with  the  internally  developed  course  management  system. 
The  center  did  choose  to  migrate  to  the  Blackboard™  server  and  implemented  many  of  the  suggested 
recommendations  for  mitigating  against  lingering  costs. 
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Abstract 

The  presentation  reviews  more  than  a decade  of  investigations  undertaken  to  determine  what  motivates 
and  what  discourages  faculty  participation  in  distance  education.  The  presenters  describe  the  evidence  that 
faculty  extrinsic  and  intrinsic  conditions  both  influence  willingness  to  participate.  The  researchers  will  also 
compare  the  findings  of  this  study  with  three  other  studies  conducted  on  faculty  motivation.  The  analysis  reveals 
that  more  recent  studies  indicate  extrinsic  motivators  are  playing  an  increasingly  important  role  in  DE.  The 
presentation  will  summarize  the  policy  implications  for  this  body  of  research. 

If  distance  education  coursework  continues  to  expand,  as  predicted  ( Hannafin.  Hannafin,  Hooper, 
Rieber  & Kini,  1996;  Texas  Higher  Education  Coordinating  Board,  2000;  Twigg,  & Oblinger,  1996;  Van 
Patten,  2000),  faculty  would  be  crucial  elements  in  the  creation  and  maintenance  of  distance  education  courses. 
Higher  education  administration  must  support  their  most  important  asset,  faculty,  so  that  faculty  maintain  both 
their  academic  positions  and  their  positions  within  their  communities  ( Chronicle  of  Higher  Education,  2001a; 
Chronicle  of  Higher  Education,  2001b;  Kezar,  2002).  Policies  that  create  motivating  conditions  for  faculty 
participation  and  that  mitigate  or  remove  inhibitors  could  sustain  and  stimulate  faculty  participation  in  DE. 

The  purpose  of  this  analysis  is  to  identify  what  conditions  have  the  greatest  influence  and  which  can  be 
manipulated  by  faculty.  A comparison  of  the  top  five  motivator  items  in  three  other  university  studies  using  a 
similar  survey,  a private  eastern  university,  a public  eastern  university,  and  a southeastern  university,  indicated 
there  were  similarities  to  some  of  the  findings  of  the  southwestern  public  university  study,  all  reflecting  faculty 
perceptions  that  the  strongest  forces  influencing  their  participation  were  intrinsic,  although  not  always  in  the 
same  order  of  priorities.  Other  findings  of  this  study  reveal  a growing  trend  towards  extrinsic  motivation 
having  a stronger  influence  on  participation  in  DE. 

Background 

Studies  prior  to  2001  reported  that  intrinsic  motivation,  a person’s  willingness  to  perform  acts  based  on 
the  internal  rewards  of  emotional  satisfaction,  was  a strong  influence  on  participation  or  nonparticipation  in 
innovation  (Betts,  1998;  Dillon  & Walsh,  1992;  Ellis,  1984;  Herzberg,  1964;  Lepper,  Keavney  & Drake,  1996; 
Iyengar  & Lepper,  1999;  Lewis,  2001;  Stephenson,  1997;  Vroorn,  1964;  Wolcott,  1997;Wolcott,  2002a; 

Wolcott,  2002b;  Wolcott  & Betts,  1999;  Wolcott  & Haderlie,  1995).  Faculty  have  reported  participating  in  DE 
for  intrinsic  rewards  over  extrinsic  rewards,  such  as  promotion  and  tenure,  grant  money,  increased  salary, 
additional  training,  or  course  releases  (Bebko,  1998;  Betts,  1998;  Brown  & Floyd,  1998;  Dillon  and  Walsh, 

1992;  Johnston,  Alexander,  Olcott  & Wright,  1995;  Schifter,  2000;  Wolcott,  1999;  Wolcott,  1997).  However, 
more  recent  studies  (Amone,  2002;  Bower,  2002;  Culp,  Riffee,  Starrett,  Sarin,  & Abrahamsen,  2001;  Gannon- 
Cook,  2003;  Twigg,  2000;  Weber,  1999)  revealed  extrinsic  rewards  were  also  motivating  DE  participation. 

Faculty  teaching  DE  courses  (as  of  the  end  of  2002)  in  the  field  of  education  in  the  United  States  earn  a 
average  salary  of  $42,000.00  for  a nine-month  contract  for  non-tenure  track  faculty,  and  $45,000.00  for  new 
tenure -track  faculty  (Chronicle  of  Higher  Education,  2003;  Johnston,  Alexander,  Conrad,  & Fieser,  2000;  Sloan 
Center  for  Asynchrononous  Learning  Environments  [SCALE],  1998).  Average  starting  salaries  for  persons  with 
undergraduate  degrees  in  business  are  about  the  same  salary  range,  $40,000.00-$42,000.00  (Wall  Street  Journal, 
2002).  Teachers  with  undergraduate  degrees  and  teaching  certificates  also  earn  salaries  in  the  same  range, 
$40,000-42,000  (Chronicle  of  Higher  Education,  2002;  Houston  Independent  School  District,  2002).  Starting 
Ph.D.  graduates  in  Computer  Science,  however,  average  around  $70,000.00  and  in  Business,  around  $60,000.00 
(American  Association  for  Higher  Education,  2001;  National  Center  for  Education  Statistics,  2001;  United 
States  Department  of  Education,  National  Center  for  Education  Statistics,  1999).  This  disparity  in  higher 
education,  with  salaries  for  doctorates  in  education  averaging  about  $20,000  per  year  less  than  Computer 
Science  and  Business,  is  reflected  in  many  universities  throughout  the  United  States  (American  Association  for 


223 


Higher  Education,  2001;  Chronicle  of  Higher  Education,  2003;  National  Center  for  Education  Statistics,  2001; 
United  States  Department  of  Education,  1997).  The  lower  salaries  for  faculty  in  Education  could  shed  light  on 
why  lack  of  compensation  incentives  could  be  de-motivating  to  faculty,  and  why  compensation  incentives  could 
make  a difference  in  motivating  faculty  members  to  participate  in  DE. 

Methodology 

To  determine  which  factors  influence  faculty  participation  in  distance  education,  we  identified  the 
survey  most  often  used  in  published  studies  to  measure  faculty  distance  education  attitudes.  At  least  eight 
institutional  research  studies  investigating  faculty  attitudes  toward  distance  education  collected  data  with  parts 
of  the  Betts  (1998)  survey  (Bebko,  1998;  Berge  & Milenburg,  2001;  Crawford  & Hunt,  1999;  Halfhill,  1998; 
Kambutu,  1998;  Montgomery,  1999;  Wolcott  & Betts,  1999);  seven  other  studies  had  similar  items  but  did  not 
replicate  the  survey  items  (Bonk,  2001;  Bower,  2002;  Byun,  2000;  Ellis,  2000;  Groves  & Zemel,  1999; 

Johnston,  2000;  Mitchell,  1999).  Only  four  of  the  studies  (Beggs,  2000;  Betts,  1998;  Gannon-Cook,  2003; 
Schifter,  2000)  published  comparable  data  for  the  same  Likert  items  to  measure  faculty  motivators  and 
inhibitors  for  distance  education  participation;  two  of  the  four  had  the  exact  same  53  items,  with  a third  study 
combining  several  intrinsic  and  extrinsic  motivating  factors  (Beggs,  2000).  The  fourth  survey  (Schifter,  2000) 
included  fifty  items  and  excluded  three  extrinsic  motivator  items  measuring  attitudes  toward  salary  increases, 
course  releases  or  other  workload  credit  for  distance  education  participation,  and  royalties  associated  with 
course  design. 

Table  one  displays  the  five  highest  ranked  Likert  items  on  each  of  the  four  studies  (Beggs,  2000;  Betts, 
1998;  Gannon-Cook,  2003;  Schifter,  2000).  The  same  five  items  had  the  highest  means  on  the  two  studies 
conducted  in  the  Southern  United  States.  All  four  of  the  studies  included  three  of  the  same  items:  personal 
motivation  to  use  technology;  ability  to  reach  new  audiences;  greater  course  flexibility  for  students.  These 
items,  while  of  some  interest,  reveal  little  about  the  underlying  factors  that  support  motivation  and  reflected 
faculty  motivation  as  a function  of  their  own  internal  values  and  were  consistent  with  earlier  studies.  Only  the 
Gannon-Cook  study  (2003)  validated  the  survey  with  a Principal  Components  Analysis  that  revealed  the 
underlying  motivational  factors  affecting  DE  participation.  Table  two  displays  the  five  highest  ranked  inhibitors 
or  de- motivators.  Table  three  displays  the  PCA  results  sustain  intrinsic  motivators  belong  to  the  strongest  factor 
but  the  next  four  factors  were  extrinsic  rewards. 

In  the  Principal  Components  Analysis  (PCA),  intrinsic  motivators  comprised  the  twelve  of  the  first  15 
(and  uppermost  ranking)  items  in  the  first  factor,  entitled  “Traditional  Staff  Service”  and  representing  internal 
drivers  to  participate  in  DE.  On  the  other  hand,  the  second  factor  and  third  factor  represented  conditions  in 
which  the  university  has  considerable  control,  were  extrinsic  motivators,  titled  “Monetary  Re  wards  “and 
“Insufficient  Rewards.”  (Factor  two  was  the  presence  of  rewards  and  factor  three,  the  lack  of  rewards).  The 
third  factor.  Insufficient  Rewards,  contained  eight  extrinsic  inhibitor  items.  The  fourth  factor.  Technical  and 
Administrative  Support,  represented  six  extrinsic  inhibitors,  as  did  the  fifth  factor.  Job  Enhancement 
Requirements,  with  three  extrinsic  motivators.  The  first  (intrinsic)  factor  comprised  19%,  the  next  four 
(extrinsic)  factors  comprised  40%,  with  the  remaining  four  (extrinsic)  factors  accounting  for  10%  of  the 
variance,  a total  of  70%  of  the  variance  accounted  for  by  the  PCA. 

Table  Three  reveals  the  items  with  highest  means,  items  with  highest  item  correlations  for  each  factor, 
and  the  five  factors  explaining  the  most  variance  (Gannon-Cook,  2003).  The  five  highest  means  were:  Personal 
Motivation  to  Use  Technology  ( Factor  one);  Ability  to  reach  new  audiences  (1);  Greater  Course  Flexibility  for 
Students  (1);  Intellectual  Challenge  (1);  and,  Opportunity  to  develop  new  ideas  (1).  The  five  items  with  the 
highest  item  correlations  for  each  factor  were:  Opportunity  to  develop  new  ideas  (Factor  one);  salary  increase 
(Factor  two);  lack  of  salary  increase  (Factor  three);  lack  of  technical  support  (Factor  four);  and,  required  by  the 
department  (Factor  five).  The  five  factors  derived  from  the  53  items  on  the  survey  that  explained  70%  of  the 
variance  were:  Traditional  staff  service  (Factor  one),  explained  18.62%  of  the  variance;  monetary  rewards 
(Factor  two),  explained  15.34%  of  the  variance;  insufficient  rewards  (Factor  three),  explainedl2%  of  the 
variance,  but  note  that  this  factor  was  the  extrinsic  inhibitor  counterpart  to  the  Factor  Two  extrinsic  motivator, 
monetary  rewards;  technical-administrative  support  (Factor  four),  explained  7.35%  of  the  variance;  and  job 
advancement  requirements  (Factor  five),  explained  5.42%  of  the  variance.  Factors  six  (Professional  Quality) 
and  seven  (Professional  and  personal  prestige)  combined  accounted  for  the  next  6%  of  the  variance,  and  Factors 
eight  (Bad  Press)  and  nine  (Personal  Benefits)  accounted  for  the  remaining  5%  of  the  variance.  All  totaled, 
intrinsic  motivators.  Traditional  staff  Service  (1)  and  Professional  Quality  (6)  combined,  accounted  for  22%  of 
the  variance;  extrinsic  motivators  accounted  for  the  remaining  48%,  with  Factors  two  and  three  accounting  for 
27%  of  the  variance. 
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DE  Survey  Response  Patterns 

The  DE  survey  patterns  indicated  that  university  faculty  perceived  the  strongest  forces  that  would 
influence  their  participation  were  intrinsic  (19%  of  variance)  and  extrinsic  motivators  (35%),  not  inhibitors 
(15%),  yet  the  responses  to  the  DE  survey  inhibitor-item  questions  contained  higher  means.  There  were  some 
research  findings  that  indicated  inhibiting  or  negative  survey  items  can  receive  stronger  participant  responses 
(Cuban,  1999;  Culp,  2001;  Johnston,  Alexander,  Conrad,  & Fieser,  2000;  Kaufman,  1992;  Lepper  & Keavney, 
1996;  Noble  1996;  Postman,  1997;  Robinson,  1995).  Apparently,  faculty  may  feel  strongly  about  some  of  the 
topics,  but  not  have  conscious  knowledge  of  those  feelings.  Faculty  also  may  feel  strongly  about  not  being 
included  in  important  institutional  decisions,  and  because  of  this,  may  be  inclined  to  respond  negatively,  or  not 
at  all,  to  inhibiting  questions  that  ask  about  what  is  lacking  or  not  being  done  at  the  institution.  The  inhibiting 
questions  act  as  double -negatives,  demonstrating  respondents’  assent  that  the  absence  of  certain  items  will  deter 
or  prevent  participation  in  DE.  So,  factor  items,  such  as,  lack  of  salary  increase,  credit  and  promotion, 
recognition  and  awards,  release  time,  and  increased  faculty  workload,  could  all  have  more  decision-making 
weight  in  the  minds  of  the  respondents,  than  indicated  by  the  survey  responses.  Yet,  the  inhibitor  factor  items 
do  rank  third,  fourth,  sixth,  and  eighth  in  the  nine  PCA  factor  scale,  giving  some  consideration  to  the  items  in 
these  factors. 


Discussion 

Lack  of  incentives  has  become  an  increasing  barrier  to  institutional  growth  in  DE.  Studies,  such  as 
those  conducted  at  higher  educational  institutions  in  Pennsylvania  (Broskoske  & Harvey,  2000;  Distance 
Education  Report,  2001;  Pennsylvania  State  University,  2002),  found  that  issues  related  to  faculty  were  far 
more  significant  for  the  success  of  DE  than  technological  issues  for  the  success  of  DE.  Extrinsic  motivators, 
while  reported  in  many  studies  as  non -motivating  (Betts,  1998;  Lepper,  1998;  Schifter,  2000;  Wolcott  & Betts, 
1995;  Wolcott  & Haderlie,  1995),  are  hard  to  ignore  when  basic  physiological  needs  must  be  met.  For  example, 
hunger,  a basic  biological  need,  makes  it  necessary  to  earn  money  to  buy  food,  and  then  money  becomes  a 
specific  drive.  The  other  sets  of  ascending  needs  relate  to  achievement  and,  through  achievement,  to  the 
experience  of  psychological  growth.  For  example,  rewards  for  successful  academic  job  performance  usually 
include  more  money,  promotions,  or  course  releases  for  research  (Bonk,  2001),  so  faculty  who  teach  DE  would 
expect  to  be  rewarded  similarly,  through  salary,  promotion/tenure,  or  adjusted  workload.  However,  to  date,  this 
has  not  been  the  case  in  most  academic  institutions  (American  Association  for  Higher  Education,  2001;  Beggs, 
2002;  Longmate  & Cosco,  2002;  National  Education  Association,  2000;  Pennsylvania  State  University,  2002; 
Rockwell,  Schauer,  Fritz,  & Marx,  1999).  The  National  Education  Association  reports  that  63%  of  the  faculty 
who  teach  DE  courses  are  compensated  for  a DE  course  as  if  it  were  a traditional,  face-to-face  course  (2000). 

Several  other  studies,  such  as  one  conducted  by  the  United  States  Department  of  Education  (1997), 
support  that  incentives  do  appear  to  play  a major  role  in  faculty  decisions  regarding  participation  (American 
Association  of  Higher  Education,  2001;American  Distance  Education  Consortium,  2001;  National  Education 
Association,  2000;  Task  Force  on  Development  of  the  Technology  Workforce,  2000;  Texas  Higher  Education 
Coordinating  Board,  2000).  Wolcott  and  Betts  (1999)  examined  the  concept  of  equity  in  relation  to  the  faculty’s 
perceived  return  on  investment.  And,  when  the  exchange  was  not  equitable,  for  t ime,  etc.,  the  DE  became  a 
disincentive  to  participation.  Faculty  who  doubted  they  would  be  adequately  rewarded  cited  the  following 
reasons:  concern  for  inadequate  financial  rewards,  workload  concerns,  concerns  relating  to  research  and 
publication,  and  dis  trust  of  administrators  (Wolcott,  1997).  Lack  of  adequate  rewards  has  been  shown  to  be  a 
personal  disincentive  as  well  as  a barrier  to  institutional  development  in  DE  (United  States  Department  of 
Education,  1997).  To  date,  faculty  participation  in  DE  has  not  been  formally  rewarded  through  advancement  in 
rank,  tenure,  or  merit  pay  in  most  academic  settings  (Beggs,  2002;  Betts,  1998;  Bonk,  2001;  Compensation 
Project  Research  in  Education,  2000;  Culp,  2001;  Johnston,  Alexander,  Conrad  & Fieser,  2000;  Schifter,  2000b; 
United  States  Department  of  Education,  1997;  Wolcott,  1997;  Wolcott  2002). 

The  review  of  the  literature  on  faculty  motivation  suggested  that  potential  DE  adopters  need  to  have 
enough  time  to  become  more  comfortable  with  the  use  of  technology,  that  peer  mentoring  should  be  offered  by 
the  institution,  and  that  both  training  and  follo\wup  training  should  be  provided,  especially  during  the  initial 
personal  concerns  stages  of  adoption  (Bandura,  1982;  Beggs,  2002;  Fullan,  1991;  Fullan,  1994;  Hall  & Hord, 
1987;  Lick  & Kaufman,  2000;  Murphy,  Walker,  & Webb,  2001;  National  Council  for  Educational  Technology, 
1995;  Robinson,  1995;  Rogers,  1995;  Sherry,  1998;  Smithers  M & Spratt,  C,  1999;  Stribak  & Paul,  1998; 
Wilson,  1999).  While  questions  about  peer  mentoring  were  not  posed  in  this  study,  the  faculty  survey  responses 
referenced  the  need  for  ongoing  training,  and  for  administrative  and  peer  support.  If  the  investment  of  monies 
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for  the  implementation  of  DE  are  sufficient,  then  the  investment  of  time  to  reinforce  faculty  adoption  of  DE 
would  be  minimal  compared  to  the  emotional  security  of  faculty  new  to  DE. 

The  DE  survey  results  supported  other  studies’  research  recommendations  to  provide  ongoing 
scaffolding  of  training  for  faculty  (Bonk,  2001;  Fullan,  1991;  Johnston,  Alexander,  Conrad  & Fieser,  2000; 
National  Council  for  Educational  Technology,  1995,  Robinson,  1995;  Wolcott,  2002).  Training  should  be 
provided  more  than  once,  and  should  be  particularly  important  at  the  management-concerns  stage  of  adoption, 
when  “how  to  do  it”  workshops  provide  crucial  reinforcement  to  faculty  still  unsure  about  their  decision  to  buy 
into  DE  participation  (Bandura,  1982;  Hall  & Hord,  1987,  Lick  & Kaufman,  2000;  Robinson,  1996). 

In  Hall  and  Hord’s  (1987)  Concerns  Based  Adoption  Model  (CBAM),  it  was  recommended  that 
facilitators  visit  more  often  with  potential  adopters  on  a face-to-face  basis  to  offer  assistance  and 
encouragement.  In  their  study,  25%  of  the  respondents  ranked  personal  support  and  training  as  most  important 
to  adoption  of  an  innovation.  In  that  study,  the  findings  revealed  faculty  often  were  not  aware  of  training  offered 
by  the  institution  (1987).  It  was  recommended  that  more  training  sessions  be  held  and  that  more  advertising  be 
done  to  make  faculty  aware  of  available  training  and  that  the  institution  supported  their  efforts.  Faculty  placed  a 
high  priority  on  technical  training  and  support  in  this  survey  too,  similar  to  the  Hall  and  Hord’s  survey  (1987). 

It  appears  that  DE  can  be  successful  and  can  become  integrated  into  the  university  culture  when 
implementations,  such  as  enough  time  to  become  more  comfortable  using  technology,  peer  mentoring,  follow- 
up and  ongoing  training,  are  offered  consistently  to  faculty  and  incorporated  into  university  DE  plans  (Bandera, 
1982;  Beggs,  2002;  Fullan,  1991;  Fullan,  1994;  Hall  & Hord,  1987;  Lick  & Kaufman,  2000;  Murphy,  Walker, 

& Webb,  2001;  National  Council  for  Educational  Technology,  1995;  Robinson,  1995;  Rogers,  1995;  Schott  & 
GannonCook,  2002;  Sherry,  1998;  Smithers  & Spratt,  1999;  Stribak  & Paul,  1998;  Wilson,  1999). 

This  Survey’s  Findings  Compared  to  Three  Other  University  Survey  Findings 

A comparison  of  the  top  five  motivator  items  in  the  four  university  studies,  the  private  eastern 
university  survey,  the  public  eastern  university,  the  southern  university,  and  the  southwestern  university, 
indicated  there  were  similarities,  all  reflecting  faculty  perceptions  that  the  strongest  forces  influencing  their 
participation  were  intrinsic,  although  not  always  in  the  same  order  of  priorities.  For  example,  personal 
motivation  ranked  first  for  the  eastern  public,  southeastern  and  southwestern  public  universities,  but  third  for 
the  private  eastern  university.  Opportunity  to  develop  new  ideas  ranked  second  for  the  private  eastern  and 
public  eastern  universities,  but  fifth  for  the  southeastern  and  southwestern  public  universities.  Ability  to  reach 
new  audiences  ranked  first  for  the  private  eastern  university  and  second  for  the  southeastern  and  southwestern 
public  universities,  but  did  not  make  the  top  five  rankings  for  the  eastern  public  university.  (See  Tablel). 

What  these  findings  reflected  was  a validation  that  faculty  do  care  for  their  students  and  are  personally 
motivated,  intrinsically,  to  teach.  They  also  care  about  having  opportunities  to  develop  new  ideas,  to  improve 
their  teaching,  and  to  be  intellectually  challenged.  In  addition,  they  care  about  having  the  ability  to  reach  new 
audiences  who  might,  otherwise,  not  be  able  to  attend  college,  and  having  greater  course  flexibility  for  students. 
But  these  motivations  would  be  present,  whether  these  faculty  taught  via  DE  or  not,  because  intrinsic  motivators 
are  the  key  reasons  why  teachers  inherently  choose  the  profession  of  teaching. 

The  findings  of  the  southwestern  survey  differed  from  the  earlier  three  surveys,  however  after  first 
intrinsic  factor  of  traditional  service  (which  included  those  factors  shown  in  Table  2).  All  of  the  remaining 
factors,  save  Factor  6 (Professional  Quality)  were  extrinsic.  Interestingly,  the  earlier  three  studies  also  cited 
extrinsic  factors  too,  but  these  were  rated  lower  as  influential  to  faculty  motivation  (See  Table  2).  While  the 
extrinsic  inhibiting  factors’  sequence  varied  among  the  four  studies,  there  were  a number  of  similarities,  such  as 
lack  of  technical  support,  which  ranked  first  by  the  Southeastern  private  and  public  universities,  and  second  for 
the  southeastern  public  university  (it  ranked  ninth  for  the  southwestern  university).  Concern  over  faculty 
workload  ranked  first  for  the  southeastern  university,  with  the  eastern  public  and  private  universities  ranking  it 
second  (it  ranked  eighth  for  the  southwestern  university).  Lack  of  release  time  ranked  third  for  the  eastern 
public  university,  but  fourth  for  the  eastern  private  and  southeastern  universities  (it  ranked  seventh  for  the 
southwestern  university).  The  highest  inhibiting  factor  mean  in  the  southwestern  public  university  study  was  the 
lack  of  salary  increase;  second,  lack  of  merit  pay;  third,  no  credit  for  work  or  promotion;  fourth,  lack  of 
monetary  support;  and,  fifth,  lack  of  recognition.  Ranked  sixth  was  the  desire  for  royalties,  and  the  seven 
through  ten  mean  rankings  were  lack  of  recognition,  lack  of  release  time,  concern  over  faculty  workload,  and 
lack  of  technical  support.  Despite  the  findings  of  the  earlier  studies  indicating  the  highest  means  for  intrinsic 
motivators,  those  studies  revealed  there  were  indicators  cited  in  those  studies  that  did  cite  some  of  the  extrinsic 
factors  in  their  studies.  Those  extrinsic  factors  surfaced  again  in  the  later,  southwestern  public  university  study, 
but  this  time  indicating  a growing  trend  by  faculty  to  choose  extrinsic  over  intrinsic  factors  to  motivate  their 
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participation  (or  nonparticipation)  in  DE. 


Reading  in-between  the  Lines  of  the  Survey  Results 

Research  indicated  that  early  adopters  of  DE,  particularly  computer  and  Internet-based  DE,  were 
intrinsically  motivated  to  participate  in  DE  (Betts,  1998;  Dillon  & Walsh,  1992;  Olcott  & Wright,  1995;  Rogers, 
1995;  Wolcott,  1995;  Wolcott  & Haderlie,  1995).  Feelings  of  accomplishment  and  satisfaction  were  enough 
reward  for  leading  the  way  into  innovation  for  these  electronic  pioneers.  However,  the  growth  of  DE  and  the 
pace  to  which  it  has  accelerated  so  rapidly  have  put  tremendous  pressures  on  universities  and  the  pioneers  in  DE 
delivery.  Today’s  DE  faculty  slump  under  the  burden  of  too  many  e-mails  and  little  or  no  help  from  teacher 
assistants  or  office  staff.  Surprisingly,  many  faculty  still  seem  willing  to  consider  taking  on  such  a burden  if 
they  perceive  their  university  is  interested  in  supporting  their  efforts  (Beggs,  2001;  Bonk,  Kirkley,  Hara  & 
Dennen,  2001;  Bower,  2002;  Byun,  Hoseung,  Paul,  Hallett,  Karen,  & Essex,  2000;  Johnston,  Alexander, 

Conrad  & Fieser,  2000;  Mitchell,  1999;  Stevenson,  2001).  But  these  faculty  also  understand  that  time  is  a very 
precious  commodity  for  them,  so  time  spent  on  DE  will  likely  take  time  away  from  some  other  priority,  such  as 
research.  Extrinsic  motivation  has  been  claimed  to  be  ineffective  as  a motivator,  but  a number  of  the  research 
studies  that  reported  those  results  either  had  small  numbers  of  responses  (Herzberg,  1964;  Lepper,  1988),  or 
were  conducted  more  than  two  years  ago  (Betz,  1998;  Lepper,  1988,  1992,  1996,  1997;  Maslow,  1970;  Schifter, 
2000;  Wolcott,  1999).  The  newer  research  studies  indicate  a strong  trend  towards  extrinsic  motivators  as  being 
crucial  to  faculty  decisions  to  participate  (or  not)  in  DE  (Bonk,  2001;  Bower,  2003;  Culp,  2001:  Gannon  Cook, 
2003;  Johnston,  2001;  O’Quinn,  2001;  Wolcott,  2002b). 

Implications  for  Practice  and  Policy  for  Institutions  of  Higher  Education 

The  most  important  influences  remain  intrinsic  motivators,  factors  that  the  university  cannot  control. 
On  the  other  hand,  university  policies  can  be  crafted  to  enhance  the  extrinsic  motivational  factors  of  monetary 
rewards,  insufficient  rewards,  technical-administrative  support,  and  mandating  participation.  Mandating 
participation  is  a poor  policy  choice  since  job  satisfaction  and  job  stress  are  directly  related  to  faculty  control 
over  the  job  and  job  tasks,  the  latter  could  lead  to  a corresponding  drop  in  job  satisfaction  and  costly  increased 
faculty  turnover. 

The  southwestern  university  study  provided  a better  understanding  of  faculty  needs  and  concerns  with 
respect  to  distance  education;  and  provided  information  that  can  be  used  for  distance  education  faculty 
development  programs  and  distance  education  policy  revisions  at  the  university  studied  in  the  survey. 
Universities  that  are  encouraging  voluntary  participation  in  DE  and  are  valuing  their  faculty  with  extrinsic 
motivators  along  with  administrative  support,  are  faring  better  with  employee  retention  and  ongoing  DE 
participation. 

If  faculty  see  the  commitment  to  DE  is  there,  evidenced  by  multiple  examples  of  what  the 
administration  is  willing  to  do  to  support  their  commitment,  such  as  technical  assistance,  course  releases,  and 
salary  increases,  then  faculty  members  might  be  more  willing  to  participate  in  DE.  The  support  of  the 
administration  could  demonstrate  that  authentic  participation  is  actually  occurring  within  the  university  and  is 
not  mere  rhetoric  (Anderson,  1998;  Beggs,  2002;  Bonk,  2001; ).  Authentic  participation  by  administrative  role- 
modeling lets  the  faculty  know  there  is  "buy-in";  but,  more  importantly,  it  conveys  the  message  that  the 
innovation  is  beneficial  to  both  the  university  and  to  the  faculty  (Anderson,  1998;  Clark  & Kaufman,  2000; 
Herzberg,  1987;  Stribiak  & Paul,  1998;  John-Steiner,  Weber,  1999).  The  university  administration  must  be  an 
integral  part  of  the  faculty  DE  team,  leading  to  success,  not  just  presiding  overDE  in  a top-down  mode. 
Authentic  participation  by  administration  creates  an  environment  conducive  to  teambuilding,  nurturing  and 
collaboration  that  extends  throughout  the  university  (Anderson,  1998).  “The  culture  and  obligation  of  the 
university  rewards  system  must  reflect  (administrative  support),  not  in  rhetoric,  but  in  reality”(Hardi,  2000, 
available  on-line).  Faculty  need  to  feel  valued. 

Value  is  intrinsic,  but  society  places  a value  on  value  by  assigning  price  tags  to  even  the  most  modest 
of  living  accommodations.  Therefore,  extrinsic  motivators,  such  as  stipends,  merit  pay,  and  grants  could  help 
reinforce  the  university’s  acknowledgement  of  value  to  faculty  who  participate  in  DE. 

It  would  be  interesting  to  follow  surveys  on  DE  participation  over  the  next  several  years  to  see  if  this 
trend  continues,  but  judging  from  the  more  recent  findings  of  the  last  two  years  (Beggs,  2002;  Bonk,  2001; 

Culp,  Riffee,  Starrett,  Sarin,  Abrahamsen,  2001;  Distance  Education  Report,  2001;  Hunt  & Crawford,  2001; 
Johnston,  Alexander,  Conrad  & Fieser,  2000;  Kirk  & Shoemaker,  1999;  McKenzie,  Mims,  Bennett,  Waugh, 
2000;  Rockwell,  Schauer,  Fritz  & Marx,  1999;  Schott,  2002;  Southeast  Missouri  State  University,  2002; 
Southern  Utah  University,  2002;  Wilson,  1999;  Wolcott,  2002),  and  it  is  likely  that  it  will,  institutions  will  need 
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to  accommodate  these  extrinsic  motivation  needs  or  face  the  risk  of  faculty  attrition  and  challenges  to  the 
delivery  DE  programs. 

The  growing  need  for  academe  to  adapt  to  electronic  delivery  is  immediate,  “not  just  to  avoid 
extinction,  but  to  actively  cultivate  opportunity”  (Kiernan,  2002,  p.54).  Studies,  such  as  this  one,  could  help 
with  assessments  of  which  factors  could  motivate  faculty  to  deliver  these  e-courses.  “Academe  must  adapt  its 
approaches  to  governance,  too,  to  react  more  nimbly  to  technological  changes. . .consultation  and  consensus- 
building are  important  in  shared  governance,  in  part  to  make  sure  that  decisions  are  made  thoughtfully... It’s 
important  that  all  members  of  the  (academic)  community  are  involved”  (2002,  p.54). 
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Table  1 . Highest  Item  Means  Motivator-Inhibitors  from  Urban  Universities 


1 

Eastern  Private  (Betts, 
1998) 

Eastern  Public 
(Schifter,  2000)* 

Southeastern 
Public  (Beggs, 
2000) 

Southwestern  Public 
(SP;  Gannon-Cook, 
2003) 

1 

Ability  to  reach  new 
audiences  (1) 

Personal  motivation  to 
use  technology  (1) 

Personal  motivation 
to  use  technology  (1) 

Personal  Motivation  to 
Use  Technology  (1) 

2 

Opportunity  to  develop 
new  ideas  (1) 

Opportunity  to  develop 
new  ideas  (1) 

Ability  to  reach  new 
audiences  (1) 

Ability  to  reach  new 
audiences  (1) 

3 

Personal  motivation  to 
use  technology  (1) 

Opportunity  to  improve 
my  teaching  (1) 

Greater  course 
flexibility  for 
students  (1) 

Greater  Course  Flexibility 
for  Students  (1) 

4 

Intellectual  challenge  (1) 

Opportunity  to  diversify 
program  offerings  (1) 

Intellectual 
challenge  (1) 

Intellectual  Challenge  (1) 

5 

Overall  job  satisfaction 
( 1) 

Greater  course  flexibility 
for  students  (1) 

Opportunity  to 
develop  new  ideas 

All 

Opportunity  to  develop 
new  ideas  (1) 

*Three  items  excluded:  Salary  increase,  course  release,  and  royalties 


Table  2.  Comparison  of  Four  Universities’  Top  Inhibitors 


■ 

Private  Urban 
Eastern 

Public  Urban 
Eastern 

Southeastern  Public 

Southwestern 

Public* 

1 

Lack  of  tech  support 
by  inst. 

Lack  of  tech  support 
by  inst 

Concern  over  Faculty 
workload 

Lack  of  Salary 
Increase  (Factor  3) 

2 

Concern  about 
faculty  workload 

Lack  of  Release  Time 

Lack  of  Tech  Support 

Lack  of  merit  pay  (3) 

3 

Lack  of  release  time 

Concern  about  faculty 
workload 

Lack  of  release  time 

No  credit  for  work  or 
promotion  (3) 

4 

Lack  of  grants  for 
materials/  expenses 

Lack  of  grants  for 
materials/  expenses 

Concern  over  quality 
of  courses 

lack  of  monetary 
support  (3) 

5 

Concern  over  quality 
of  courses 

Concern  over  quality 
of  courses 

Lack  of  DE  Training 

Lack  of  recognition 

(3) 

*Gannon-Cook,  2003,  p.193. 


Table  3.  Items  with  highest  means,  items  with  highest  item  correlations  for  each  factor,  and  five  factors 
explaining  the  most  variance  (Gannon -Cook,  2003) 


Items  by  highest  means  of 
fifty  items 

Five  items  with  highest  item 
correlations  for  each  (factor) 

Five  factors  derived  from  53  items 
explaining  70%  of  the  variance 

Personal  Motivation  to  Use 
Technology  (1) 

Opportunity  to  develop  new  ideas  (1) 

Traditional  staff  service 

Ability  to  reach  new  audiences 
(1) 

Salary  increase  (2) 

Monetary  rewards 

Greater  Course  Flexibility  for 
Students  (1) 

Lack  of  salary  increase  (3) 

Insufficient  rewards 

Intellectual  Challenge  (1) 

Lack  of  technical  support  (4) 

Technical-administrative  support 
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Opportunity  to  develop  new 

Required  by  dept  (5) 

Job  advancement  requirements 

ideas  (1) 
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Helping  Learners  Gain  Diagnostic  Problem  Solving  Skills:  Specific  Aspects 
of  the  Diagnostic  Pathfinder  Software  Tied  to  Learning  Outcomes 

Jared  Danielson 
Holly  Bender 
Eric  Mills 
Pamela  Vermeer 
Vanessa  Preast 
Iowa  State  University 

Introduction 

Problem  solving  is  of  critical  importance  in  many  disciplines.  In  medicine,  the  clinician’s  ability  to 
arrive  at  the  correct  diagnosis  often  means  the  difference  between  life  and  death.  Despite  its  importance  and  a 
significant  amount  of  research  regarding  how  to  improve  problem  solving,  few  unambiguous  answers  have 
emerged  for  promoting  problem  solving  learning. 

This  paper  follows  up  on  recent  published  findings  (Danielson,  Bender,  Mills,  Vermeer,  & Lockee, 
2003)  concerning  the  Diagnostic  Pathfinder  (DP),  a software  learning  tool  designed  to  help  clinical  pathology 
students  improve  their  ability  to  solve  diagnostic  problems.  That  paper  described  the  functionality  of  the  DP, 
and  showed  that  using  the  DP  for  homework  and  lecture  improved  students’  grades  on  a case-based  final  exam 
by  a full  letter  grade.  Very  little  was  discussed  in  that  paper  as  to  how  characteristics  of  the  software  supported 
the  gains  that  were  seen.  In  the  two  years  since  that  report  was  written,  the  DP  has  been  used  by  640  more 
students  at  five  college  of  veterinary  medicine.  Quantitative  learning  gains  similar  to  those  reported  in  2003 
have  been  observed  (as  yet  unpublished)  in  two  other  settings.  In  this  paper  we  turn  our  attention  to  explaining 
those  gains,  both  theoretically,  and  in  terms  of  the  qualitative  data  we  have  been  able  to  collect  from  learners 
over  the  past  two  years.  Our  goal  is  to  associate  those  gains  with  specific  characteristics  of  the  DP  by  and 
meaningfully  categorizing  and  characterizing  thousands  of  comments  from  students  who  used  the  DP  in  a 
number  of  different  settings,  and  to  use  those  comments  to  illustrate,  from  the  students’  perspective,  how  using 
the  DP  accomplished  what  it  did.  These  ideas  will  be  tied  to  current  theory  regarding  the  teaching  and  learning 
of  problem  solving. 


Defining  Problem  Solving 

One  difficulty  in  building  upon  current  problem  solving  research  is  that  researchers  in  various  fields 
study  problem  solving  for  differing  purposes  and  define  problem  solving  in  different  ways.  Even  within 
individual  domains,  the  conceptual  waters  are  muddy.  In  the  domain  of  cognitive  psychology,  for  example, 
where  problem  solving  has  received  considerable  attention,  the  heart  of  the  issue  seems  to  lie  in  whether  to 
define  problem  solving  in  terms  of  the  “gap”  between  what  the  problem  solver  knows  and  what  he/she  must 
figure  out  to  solve  a problem  (Wenke  & Frensch,  2003),  or  in  terms  of  measurable  characteristics  of  the 
problem  solving  task  (Quesada,  Kintsch,  & Gomez,  in  press).  Each  class  of  definitions  is  both  useful  and 
problematic.  Defining  problem  solving  purely  in  terms  of  addressing  the  gap  between  what  the  learner  knows 
and  what  he/she  needs  to  know  implies  that  literally  any  task  can  constitute  problem  solving  (or  not),  depending 
on  the  interaction  between  the  problem  solver  and  the  presented  problem.  This  state  of  affairs  makes  it  difficult 
to  provide  “controlled,”  yet  authentic,  instances  of  problem  solving  for  laboratory  study.  At  the  same  time,  the 
skill  that  problem  solvers  employ  when  they  address  gaps  in  their  own  knowledge  seems  a worthy  object  of 
study,  and  it  seems  short-sighted  to  ignore  this  phenomenon  simply  because  it  is  difficult  to  control  for  in  a 
laboratory  setting.  As  reiterated  by  Ericsson  (2003),  it  has  long  been  recognized  that  one  potential  pitfall  of 
experimental  science  is  that  simple  and  convenient  forms  of  inquiry  can  prove  inadequate  for  studying  complex 
phenomena. 

If  the  definition  of  problem  solving  is  unclear  in  the  fie  Id  of  psychology,  it  is  more  so  across  many  of 
the  disciplines  that  attempt  to  teach  students  how  to  solve  problems.  While  some  researchers  in  various  teaching 
fields  (such  as  biology  (M.  U.  Smith,  1991a)  and  chemistry  (Bodner,  1991))  have  made  laudable  efforts  to  put 
forward  unified  definitions  of  problem  solving,  one  disturbingly  prevalent  practice  seems  to  be  to  discuss 
problem  solving  in  terms  of  whatever  experts  do  in  any  given  field  when  confronted  with  a problem  that  non- 
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experts  can’t  solve.  Researchers  who  do  this  face  the  problems  embodied  in  both  of  the  definitions  put  forward 
by  cognitive  psychologists  without  reaping  any  of  the  benefits  of  either;  problem  solving  thereby  ends  up  being 
neither  defined  by  task  characteristics  nor  by  problem  solver  characteristics,  and  hence  literally  can  mean 
almost  anything  to  anybody.  (This  problem  can  be  observed  by  perusing  the  chapters  of  Smith’s  (Ed.)  Toward  a 
Unified  Theory  of  Problem  Solving:  Views  From  the  Content  Domains  (M.  U.  Smith,  1991b).) 

Researchers  in  the  domain  of  instructional  design  generally  have  committed  to  a gap-based  definition 
of  problem  solving  (e.g.  (Gagne,  Briggs,  & Wager,  1992;  Jonassen,  2000;  P.  L.  Smith  & Ragan,  1999)).  Task- 
specific  distinctions  are  considered  (perhaps  best  illustrated  in  Jonassen’ s proposed  problem-solving  typology 
(Jonassen,  2003)),  but  the  a fundamental  characteristic  of  addressing  the  unknown  in  some  way  remains  a key 
component  of  problem  solving.  Smith  and  Ragan’s  (1999)  definition  of  problem  solving,  as  “the  ability  to 
combine  previously  learned  principles,  procedures,  declarative  knowledge,  and  cognitive  strategies  in  a unique 
way  within  a domain  of  content  to  solve  previously  unencountered  problems’’  (p.  132),  is  typical.  For  the 
purpose  of  our  discussion,  we  will  align  ourselves  with  this  definition  because  it  most  closely  characterizes  the 
phenomenon  we  wish  to  discuss.  Specifically,  for  the  purposes  of  this  proposal,  problem  solving  occurs  when  a 
person  employs  existing  knowledge  and  skills  to  achieve  a goal  state  that  he  or  she  never  before  had  achieved. 

If  successful,  the  problem  solving  process  results  in  the  learning  of  new  knowledge/skills  relevant  to  solving  the 
problem  that  was  encountered,  and  enables  the  problem  solver  to  deal  with  similar  future  situations  without 
having  to  go  through  the  problem  solving  process  again.  How,  then,  is  an  instructional  designer  to  improve 
individuals’  abilities  to  do  what  they  don’t  know  how  to  do?  That  is  the  challenge  we  address  here.  To  provide  a 
framework,  we  will  first  discuss  diagnostic  problem  solving  from  the  perspective  of  medicine. 

Medical  Diagnostic  Problem  Solving 

Medical  problem  solving  literature  lacks  consensus  regarding  the  processes  and  knowledge  structures 
that  contribute  to  diagnostic  problem  solving.  This  is  partly  due  to  the  fact  that,  as  discussed  earlier,  definitions 
of  the  term  “problem  solving’’  vary  and  the  term  generally  tends  to  be  used  synonymously  with  “expert 
performance.” 

One  perspective  on  medical  problem  solving  generally  can  be  characterized  by  Schmidt,  Norman,  and 
Boshuizen’s  (1990)  proposed  stage  theory  of  clinical  problem  solving.  They  suggested  that  clinicians  employ 
four  stages  of  reasoning,  which  build  upon,  but  do  not  replace,  each  other.  Stage  1 involves  the  development  of 
elaborate  causal  knowledge  networks.  Stage  2 involves  compiling  the  elaborate  networks  into  abridged  ones.  In 
stage  3,  the  clinicians  develop  illness  “scripts,”  and  in  stage  4,  they  develop  and  use  “instance  scripts.”  Causal 
networks  in  the  first  two  stages  are  based  on  underlying  knowledge  of  pathophysiology,  although  stage  2 
networks  also  are  informed  by  observations  of  real  patients.  Stages  3 and  4 also  are  knowledge  networks,  but 
they  are  built  increasingly  on  list-like  structures  containing  information  from  real  cases  in  the  clinician’s 
memory.  The  progression  from  one  stage  to  another  occurs  as  practitioners  gain  practical  experiences  that 
contribute  to  the  cognitive  representations  of  their  knowledge.  Causal  networks  of  pathophysiology  are  not  lost, 
nor  do  they  grow  as  patient  scripts  slowly  increase.  The  result  is  that  mo  st  clinical  problem  solving  ends  up 
being  script-based,  rather  than  pathophysiology-based.  Clinicians  will  tend  to  use  the  “highesf  stage  of 
reasoning  available  to  them,  depending  on  the  availability  and  complexity  of  knowledge  they  have  in  any  given 
domain.  A clinician  who  has  experience  with  a previous  patient  that  fits  a current  patient  will  tend  to  use  that 
experience  as  an  instance  script,  whereas  in  an  unknown  situation  the  physician  will  revert  to  earlier  stages  until 
finding  one  that  best  accounts  for  the  current  problem.  This  theory  explains  the  behavior  of  expert  clinicians 
(and  seems  consistent  with  some  other  work  done  by  researchers  in  the  field  of  medical  education  (G.  J.  Groen 
& Patel,  1985;  Guy  J.  Groen  & Patel,  1991;  Patel,  Groen,  & Norman,  1991)  and  with  the  generally  accepted 
concept  of  the  automaticity  of  expertise  (Speelman,  1998;  Winn  & Snyder,  1996)).  However,  note  that  scripted 
behaviors  are  unlikely  to  entail  problem  solving  as  we  have  defined  it,  because  those  are  the  behaviors  that 
experts  use  when  addressing  problems  they  are  familiar  with  (or  feel  they  are  familiar  with),  and  not  when  they 
address  previously  unencountered  problems.  Therefore,  behaviors  in  “lower”  stages  are  more  likely  to  be 
characteristic  of  problem  solving  than  are  behaviors  “higher”  in  the  stages. 

Research  by  Bordage  and  Lemieux  (1991)  sheds  additional  light  on  the  model  proposed  above.  They 
examined  the  semantic  structures  of  experts  and  novices  as  such  structures  relate  to  diagnostic  thinking.  Their 
study,  based  on  structural  semantic  theory,  provided  clinical  problems  for  a variety  of  experts  and  novices  and 
examined  their  solutions  in  terms  of  semantic  relationships.  These  semantic  relationships  were  used  to  reveal 
connections  between  the  concepts  that  were  relevant  to  solving  the  clinical  problem.  Each  relationship 
identified  was  referred  to  as  a “semantic  axis.”  It  was  found  that  “the  more  diversified  the  diagnostician’s 
network  of  semantic  axes,  the  better  was  his  or  her  diagnostic  accuracy.”  In  other  words,  the  greater  the  number 
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of  valid  abstract  relationships  the  subjects  found  between  bits  of  information  in  the  problem,  the  more  accurate 
were  their  diagnoses.  The  number  of  semantic  relationships  for  each  case  was  not  necessarily  related  to  the 
number  of  words  used  to  describe  the  analysis.  Therefore,  while  an  expert  might  appear  to  arrive  at  a diagnosis 
quickly  and  without  much  thought  (i.e.,  with  very  few  words),  a semantic  analysis  would  reveal  that  the  expert 
has  understood  and  identified  a great  number  of  valid  relationships  between  the  data  without  spelling  them  out 
as  such.  Bordage  (1994)  later  examined  his  and  Lemieux’s  findings  in  terms  of  their  implications  for  arriving  at 
accurate  diagnoses.  In  his  words,  “The  most  accurate  diagnosticians,  whether  students  or  specialists,  are  those 
who  have  the  most  diversified  sets  of  semantic  axes  (those  who  have  elaborated  or  compiled  structures)  and 
who  organize  symptoms  an  signs  into  coherent  systems  of  relationships  of  abstract  qualities,  and  thus 
demonstrate  a broader  and  deeper  representation  and  understanding  of  the  problem  (p.  884).”  Several  related 
studies  (Cholowski  & Chan,  1992;  Stevens,  1991;  Stevens,  Ikeda,  Casillas,  Palacio-Cayetano,  & Clyman,  1999) 
appear  to  support  Bordage’s  assertion.  While  we  are  reluctant  to  imply  a false  dichotomy  between  the 
perspectives  proposed  by  these  researchers,  Bordage’s  statement,  in  comparison  to  Schmidt  et  al.’s  (1990) 
assertion  that  “expert  clinical  reasoning  is,  to  a large  extent,  based  on  the  similarity  between  the  presenting 
situation  and  some  previous  patient  available  from  memory”  (p.  617),  highlights  one  of  the  fundamental 
tensions  that  has  characterized  this  body  of  literature.  We  feel  that  the  evidence  now  available,  including  our 
own  research,  tends  to  support  Bordage’s  implication  that  to  truly  become  expert,  students  must  become  expert 
at  associating  relevant  information  within  a case,  rather  than  or  in  addition  to  comparing  cases.  We  will  see  that 
this  approach,  as  embodied  in  the  DP,  is  associated  by  our  students  with  an  increase  in  their  ability  to  solve 
diagnostic  problems. 


Cognitive  Tools 

The  term  cognitive  tool  has  been  used  in  a number  of  related  ways  over  the  past  decade  and  a half  (see, 
for  example,  (Jonassen,  2003;  Lajoie,  Azevedo,  & Fleiszer,  1998;  Robinson,  1999;  Salomon,  1988).  Salomon 
(1988)  presented  the  cognitive  tool  as  a computer-based  tool  using  or  modeling  an  expert  approach  to  a given 
process.  He  referred  to  the  learner’s  interaction  with  cognitive  tools  as  “AI  in  reverse.”  Whereas  the  goal  of 
traditional  artificial  intelligence  (AI)  is  to  lead  a computer  to  emulate  the  cognitive  processes  of  people, 

Salomon  suggested  that  learners’  interactions  with  specifically  designed  software  tools  would  lead  them  to 
acquire  cognitive  skills  or  strategies  embodied  in  the  software.  Jonassen  (2003)  expanded  the  cognitive  tool 
concept  as  follows: 

Cognitive  tools  are  any  technologies  that  engage  and  facilitate  specific  cognitive  activities. 

They  amplify  the  learners’  thinking  by  enabling  learners  to  represent  what  they  know  using 
different  representational  formalisms.  As  knowledge  representation  formalisms,  cognitive 
tools  are  premised  on  the  idea  that  humans  learn  more  from  constructing  and  justifying  their 
own  models  of  systems  than  from  studying  someone  else’s  (p.  372). 

In  addition  to  the  intuitive  benefit  of  making  one’s  own  thinking  apparent,  Jonassen  points  out  that  that 
such  a tool  should  also  reduce  the  significant  demand  that  complex  problem  solving  places  on  working  memory 
by  harnessing  the  computer’s  ability  to  remember  and  organize.  By  this  definition,  the  Diagnostic  Pathfinder 
(DP)  is  a cognitive  tool.  The  DP  supports  the  learning  of  problem  solving  by  presenting  students  with  problems 
and  requiring  them  to  address  those  problems  within  the  framework  of  the  software.  This  paper  does  not 
accommodate  a detailed  description  of  how  the  Diagnostic  Pathfinder  functions.  Such  a description  is  provided 
elsewhere  (Danielson  et  al.,  2003).  In  brief,  through  a series  of  interactions,  the  DP  presents  a patient  case  that 
includes  history,  signalment,  physical  exam,  and  laboratory  data,  and  then  requires  students  to  identify  all 
abnormal  laboratory  data  and  communicate  their  diagnostic  reasoning  by  organizing  those  data  into  a diagnostic 
path.  The  diagnostic  path  displays  the  students’  diagnostic  reasoning  in  an  outline  form  of  propositions  that 
relate  changes  in  laboratory  data  to  the  corresponding  disease  or  physiologic  processes  occurring  in  the  patient. 
After  students  commit  to  a diagnosis,  a diagnostic  path  created  by  a faculty  clinical  pathologist  is  revealed  for 
immediate  comparison.  This  comparison  allows  both  students  and  faculty  to  see  the  rationale  used  by  the  other 
when  analyzing  a patient’s  laboratory  data. 


Methods  and  results 


Subjects 

Between  the  Spring  of  2002  and  the  Fall  of  2004,  the  DP  was  used  to  teach  eight  semesters  of  Clinical 
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Pathology  at  five  colleges  of  veterinary  medicine.  A total  of  640  students  participated  in  these  classes,  with 
roughly  70%  of  those  students  being  female  and  30%  male.  Other  than  one  small  pilot  course  of  five  students, 
the  smallest  class  contained  42  students,  with  the  largest  containing  126,  and  the  average  class  size  being  80 
students.  All  students  participating  in  these  courses  were  asked  to  complete  a questionnaire  regarding  their 
experience  with  the  DP  software.  Five  hundred  and  forty  three  students  completed  the  questionnaire.  By  class, 
this  response  rate  varied  from  46%  (the  lowest)  to  100%  (the  highest),  with  the  overall  response  rate  being  84% 


Procedures 

Students  at  the  participating  institutions  used  the  DP  to  complete  case-based  homework  assignments 
and  prepare  for  exams.  The  number  of  DP  cases  assigned  to  each  student  using  the  DP  varied  from  institution  to 
institution,  with  the  smallest  number  being  6,  and  the  greatest  number  being  93.  Curricular  approaches  at  the 
institutions  varied  as  well,  though  all  but  one  of  the  institutions  employed  what  might  loosely  be  described  as  a 
traditional  medical  curriculum  in  which  students  received  lectures  in  Clinical  Pathology  interspersed  with 
laboratories  involving  the  discussion  of  laboratory  data  for  specific  animal  cases.  The  other  institution,  which 
only  represents  five  of  the  students  surveyed,  uses  a curriculum  that  mixes  several  approaches,  including 
traditional  strategies,  collaborative  learning,  and  problem  based  learning  (PBL). 

Instruments 

The  full  questionnaire  upon  which  the  findings  for  this  study  are  based  is  found  elsewhere  (Danielson 
et  ah,  2003).  The  questionnaire  was  designed  to  determine  the  students’  perceptions  of  the  DP’s  clarity  (or 
usability),  feasibility,  and  impact  on  learning.  Responses  to  the  items  dealing  with  feasibility  and  clarity  have 
changed  systematically  over  time  as  the  software  has  been  debugged  and  various  changes  have  been  made  in 
navigation,  etc.  We  will  not  explore  those  responses  in  this  paper.  Rather  we  will  focus  specifically  on  the  items 
designed  to  reveal  the  students’  perceived  impact  of  the  DP  on  learning.  Because  the  software’s  core  learning 
interaction  has  remained  largely  unchanged,  a comprehensive  analysis  of  the  students’  reaction  to  the 
instructional  attributes  of  the  software  can  be  performed  meaningfully.  The  questionnaire  items  having  to  do 
specifically  with  learning  outcomes  are:  6,  7,  12,  16,  and  17,  and  are  found  in  Table  1.  Other  items,  particularly 
those  intended  to  measure  enjoyment  or  ease  of  use,  can  arguably  be  hypothesized  to  indicate  learning  gains  as 
well,  at  least  indirectly.  However,  these  items  are  also  closely  tied  to  software  feasibility  factors,  such  as 
computer  bugs,  network  problems  etc.,  so  we  will  not  discuss  them  here.  In  addition  to  the  results  of  items  6,  7, 
12,  16  and  17,  we  will  examine  responses  to  the  survey’s  open-ended  questions,  many  of  which  clarify  the 
students’  general  indications  of  their  response  to  the  software’s  affect  on  learning.  Those  questions  are  as 
follows: 

23.  For  questions  above  that  you  ranked  particularly  negatively,  please  indicate  why  here. 

24.  What  are  the  things  you  like  most  about  using  the  DP? 

25.  What  are  the  things  you  like  least  about  using  the  DP? 

26.  What  would  you  change  about  the  DP  if  you  could? 

27.  Any  additional  comments  you'd  like  to  make  about  the  DP: 

28.  If  you  used  the  DP  for  less  than  20%  of  your  cases,  why  did  you  choose  not  to  use  it? 

Data  analysis  procedures 

Descriptive  statistics  were  calculated  for  responses  to  survey  Likert  items  across  all  respondents  by 
institution.  Open-ended  responses  were  analyzed  initially  by  one  of  the  primary  researchers  to  reveal  broad 
trends  in  the  data.  The  responses  were  then  codified  according  to  those  trends,  and  recorded  in  an  access 
database  by  a research  assistant.  The  coding  was  then  reviewed  and  corrected  as  necessary  by  one  of  the 
primary  researchers. 


Presentation  and  Analysis:  Likert  Items 

Table  1 reports  student  responses  to  the  Likert  items  by  institution  and  year.  Students  at  all  institutions 
generally  indicated  that  the  knowledge  or  behaviors  identified  in  items  6,  7,  12,  16,  and  17  were  enhanced  by 
DP  use.  Item  6 was  intended  to  measure  perceived  completeness  - i.e.,  how  many  data  abnormalities  students 
accounted  for.  Item  7 was  intended  to  measure  whether  or  not  students  felt  that  DP-use  affected  the  precision  of 
their  diagnostic  rationale,  and,  if  so,  whether  the  effect  was  positive  or  negative.  As  seen,  students  generally  felt 
that  DP-use  made  their  rationale  more  precise.  Items  12  and  17  were  intended  to  function  as  general  indicators 
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of  the  students’  appraisal  of  the  DP’s  overall  value  to  them  as  a learning  tool.  As  seen  by  responses  to  those 
items,  students  generally  found  the  learning  value  to  be  quite  high.  Finally,  item  16  was  intended  to  get  a 
general  sense  of  the  DP’s  affect  on  the  students’  ability  to  organize  the  data  relevant  to  solving  a particular  case. 
Again,  as  seen,  the  students  generally  indicated  that  the  DP’s  affect  on  their  ability  to  organize  data  was 
positive. 


Table  1 Mean  responses  by  item  number,  institution  and  year: 


Institution/Yr 

Mean  Median 

Max 

Min 

Sample  Size 

6.  Using  the  DP  made  me  account  for  more  lab  data  than  I otherwise  would  have  accounted  for. 

less  same  more 

123456789  10  NA 

Virginia  Tech  2002 

8.7 

9 

10 

5 

86 

Iowa  State  2002 

8.5 

9 

10 

2 

96 

Iowa  State  2003 

8.4 

9 

10 

5 

68 

Wisconsin  2003 

8.1 

8 

10 

5 

42 

Wisconsin  2004 

8.2 

8 

10 

5 

65 

California  Davis  2003 

7.7 

8 

10 

1 

55 

California  Davis  2004 

8.6 

9 

10 

2 

124 

Guelph  2003 

8.6 

9 

9 

8 

5 

7.  Using  the  DP  made  my  diagnostic  paths  more  precise  than  they  would 

less  same  more 

123456789  10  NA 

have  been  otherwise. 

Virginia  Tech  2002 

8.7 

9 

10 

4 

86 

Iowa  State  2002 

8.1 

9 

10 

1 

96 

Iowa  State  2003 

8.3 

9 

10 

1 

68 

Wisconsin  2003 

7.8 

8 

10 

3 

42 

Wisconsin  2004 

8.3 

8 

10 

3 

65 

California  Davis  2003 

7.2 

7 

10 

3 

55 

California  Davis  2004 

7.8 

8 

10 

2 

124 

Guelph  2003 

8.0 

8 

10 

6 

5 

12.  The  DP  makes  doing  my  Clinical  Pathology  homework  more  worthwhile  than  similar  paper-based 

assignments. 

definitely  not  absolutely 
123456789  10  NA 

Virginia  Tech  2002 

8.6 

9 

10 

1 

86 

Iowa  State  2002 

7.9 

9 

10 

1 

96 

Iowa  State  2003 

8.6 

9 

10 

3 

68 

Wisconsin  2003 

6.8 

7 

10 

1 

42 

Wisconsin  2004 

7.4 

8 

10 

2 

65 

California  Davis  2003 

6.1 

6 

10 

1 

55 

California  Davis  2004 

7.3 

8 

10 

1 

124 

Guelph  2003 

8.8 

9 

10 

7 

5 

16.  Using  the  DP  helps  me  to  organize  my  thoughts  about  a case. 
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definitely  not  absolutely 
123456789  10  NA 


Virginia  Tech  2002 

8.4 

9 

10 

2 

86 

Iowa  State  2002 

8 

9 

10 

1 

96 

Iowa  State  2003 

8.6 

9 

10 

4 

68 

Wisconsin  2003 

8.3 

9 

10 

3 

42 

Wisconsin  2004 

8.2 

9 

10 

2 

65 

California  Davis  2003 

6.8 

7 

10 

1 

55 

California  Davis  2004 

7.8 

8 

10 

1 

124 

Guelph  2003 

8.8 

9 

10 

8 

5 

17.  Using  the  DP  makes  understanding  clinical  pathology... 

harder 

easier 

123456789  10  NA 

Virginia  Tech  2002 

8.3 

9 

10 

3 

86 

Iowa  State  2002 

8.2 

9 

10 

1 

96 

Iowa  State  2003 

8.9 

9 

10 

6 

68 

Wisconsin  2003 

8.5 

9 

10 

4 

42 

Wisconsin  2004 

8.4 

8.5 

10 

5 

65 

California  Davis  2003 

6.9 

7 

10 

3 

55 

California  Davis  2004 

7.9 

8 

10 

1 

124 

Guelph  2003 

8.3 

8.5 

10 

6 

5 

Presentation  and  Analysis:  Open-Ended  Responses 

We  analyzed  the  responses  to  open-ended  items  with  the  hopes  of  understanding  more  specifically, 
from  the  learners’  perspective,  what  characteristics  of  the  software  produced  the  results  we  had  measured 
empirically  in  earlier  studies . Here  we  will  not  report  all  open-ended  responses,  because  of  the  large  amount  of 
information  involved.  Also,  we  will  not  report  the  results  by  survey  item  number,  because  there  isn’t  always  a 
predictable  relationship  between  the  nature  of  the  question  that  was  asked  and  the  actual  response.  For  example, 
the  question,  “What  are  the  things  you  like  most  about  using  the  DP?”  elicited  both  of  the  following  responses: 
“It  was  great  in  getting  concepts  stuck  in  my  head.  . . ” and  “I  really  didn’t  like  it.” 

The  open-ended  responses  were  analyzed  as  follows.  One  of  the  primary  researchers  read  through  all 
the  responses  several  times,  identifying  broad  thematic  categories.  All  the  open-ended  response  data  were  then 
reviewed,  coded,  and  categorized  by  a research  assistant,  in  consultation  with  one  of  the  primary  researchers 
and  with  the  aid  of  an  Access  database.  This  process  resulted  in  several  categories  being  combined,  expanded, 
or  eliminated.  The  resulting  categories,  in  order  of  most  responses  to  fewest,  were  as  follows:  1.  General 
Response,  2.  Ease/Efficiency  of  Thinking  3.  Ease  of  Use/Convenience,  4.  Requirement  that  all  data 
abnormalities  be  typed  by  hand  and  spelled  correctly.  5.  Requirement  of  completeness  6.  Expert  feedback,  7. 
Process  of  Manipulating  Data  in  the  diagnostic  path,  and  8.  Diagnostic  path  format.  Responses  in  each  category 
were  coded  as  either  (a)  positive  comments,  (b)  negative  comments  and/or  suggested  improvements,  or  (c) 
comments  that  were  mixed  (both  positive  and  negative)  or  in  some  way  unclear.  We  then  counted  each 
response-type  by  category.  Recall  that  all  543  students  responded  to  the  questionnaire.  Because  the  Likert 
responses  were  largely  positive,  it  would  be  expected  that  responses  to  open-ended  questions  would  be  as  well. 
At  the  same  time,  because  more  open-ended  questions  were  designed  to  elicit  critical  responses  or  suggestions 
for  improvement  (items  23,  25,  26  & 28)  than  positive  responses  (item  24),  it  seems  reasonable  to  expect  a 
disproportionately  high  number  of  critical  comments. 

1.  General  Response  Category: 

This  category  was  used  to  broadly  characterize  the  overall  tenor  of  individual  respondents’  appraisal 
of  the  DP  as  a learning  tool,  considering  all  the  responses  to  the  open-ended  questions.  This  category  was 
considered  important  because  many  respondents  gave  mixed  feedback  (i.e.,  some  positive  comments  and  some 
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criticisms  or  suggestions  for  improvement).  All  the  comments  for  each  respondent  were  considered 
cumulatively  to  determine  if  the  overall  impression  of  the  software  was  positive  or  negative.  In  some  cases,  it 
was  too  difficult  to  identify  one  overall  sentiment  from  a respondent’s  open-ended  responses.  Such  respondents 
were  classified  as  “unclear_neither_both”.  Of  the  543  total  respondents,  242  made  comments  that  were  judged 
to  be  predominantly  positive,  152  made  comments  that  were  classified  as  uncle ar_neither_both,  and  20  made 
comments  that  were  predominantly  negative.  Many  of  the  responses  that  contributed  to  this  broad 
categorization  will  be  discussed  when  we  discuss  the  other  nine  specific  response  categories. 

2.  Ease/Efficiency  of  Thinking 

A number  of  open-ended  responses  referred  to  the  software’s  general  effect  on  the  way  students 
perceive  themselves  as  thinking  about  the  problems.  Of  comments  in  this  category,  178  were  positive,  2 were 
mixed,  and  5 were  negative.  The  positive  comments  ranged  from  fairly  vague  statements  such  as,  “a  great  way 
to  get  us  to  think  clinically.  . .”  to  more  specific  statements  such  as,  “helps  me  be  organized  about  my  thoughts, 
shows  me  clearly  flaws  in  my  logic  and  ability  to  reason,’’  and  “. . . 1 could  organize  my  thoughts  in  a logical 
manner.”  The  two  respondents  who  gave  mixed  responses  in  this  category  seemed  to  simultaneously  feel  that 
the  DP  was  useful,  but  that  it  didn’t  change  the  way  they  would  think  about  clinical  pathology,  or  that  it  was 
inconsistent  with  their  way  of  thinking.  One  respondent  for  example,  made  both  of  the  following  statements  in 
response  to  different  questions:  “The  DP  did  not  make  me  understand  clinical  pathology  any  better.  I still 
needed  to  use  paper  at  times  to  organize  my  thoughts  and  group  findings.”  and  “[The  DP  is]  very  repetitive  and 
good  for  learning  how  to  rank  and  place  abnormalities.”  Five  students  made  comments  suggesting  that  they  felt 
the  DP  did  little  to  help  their  thinking,  or  that  the  amount  of  effort  required  to  use  the  DP  did  not  justify  the 
benefits  that  were  derived  from  its  use.  For  example,  one  respondent  said,  “I  feel  like  you  can  breeze  through 
the  case  without  really  learning  a link  between  cause  and  effect.”  Another  said,  “I  spent  much  more  time  using 
DP  or  putting  this  stuff  on  computer  than  it  took  to  understand/make  my  problem  list  and  reasonings  on  the  lab 
data  sheets.  Not  saying  it  probably  didn’t  help  cement  things,  but  it  was  more  time  than  necessary  for  my 
understanding  (I  felt).” 

3.  Ease  of  Use/Convenience 

A number  of  respondents  referred  to  the  convenience  or  ease  of  using  the  DP.  One  hundred  and  fifteen 
respondents  made  positive  comments  in  this  category,  13  made  mixed  comments,  and  21  made  primarily 
negative  comments.  Some  positive  comments  referred  to  specific  aspects  of  the  software  that  made  study  more 
convenient,  such  as  the  ability  to  work  from  any  computer,  the  ability  to  work/save  cases  on-line,  or  the  ability 
to  save  a partially  completed  case.  Such  comments  included  “online  and  total  select”  (the  latter  referring  to  the 
ability  to  select  multiple  data  items  in  manipulating  the  data),  and  “being  on  the  computer.”  Other  positive 
comments  were  simple  generic  statements  of  convenience,  such  as  “saves  time”,  or  “easy.”  Mixed  comments 
often  had  to  do  with  student  responses  to  different  aspects  of  the  software,  for  example,  one  student  objected  to 
the  fact  that  the  software  tied  up  the  phone  line,  and  experienced  some  trouble  installing  it,  but  still  indicated 
that  the  software  “Saves  time,  its  easy.”  Other  students  indicated  that  their  perception  of  the  ease  of  use  changed 
over  time.  For  example,  one  said,  “The  first  few  cases  seemed  more  overwhelming  as  the  process  was  entirely 
new;  once  I was  familiar  with  the  program,  there  were  no  difficulties  stemming  from  the  program.”  The 
negative  comments  were  usually  tied  to  factors  inherent  in  working  with  a computer.  Some  students  did  not 
own  an  adequate  computer,  and  so  completed  their  assignments  on  lab  computers,  which  they  did  not  find 
convenient.  Other  students  expressed  a preference  for  working  without  computers. 

4.  Requirement  that  all  data  abnormalities  be  typed  by  hand  and/or  spelled  correctly 

The  DP  requires  that  students  manually  enter  names  of  all  data  abnormalities  spelled  correctly.  Forty 
three  students  only  made  positive  comments  about  this  requirement,  10  students  made  mixed  comments,  and  71 
students  made  primarily  negative  comments.  Characteristic  of  positive  comments,  one  student,  when  asked 
what  he/she  liked  best  about  the  DP  wrote:  “The  structure  of  having  to  learn  vocabulary  by  retyping.  . .”  Other 
students  made  only  positive  comments  about  this  requirement,  but  only  mentioned  partial  aspects  of  the 
process.  For  example,  a number  of  students  mentioned  that  they  liked  being  told  the  correct  spelling  of  data 
abnormalities  after  three  tries.  Most  comments  in  this  category  emphasized  that  the  primary  benefit  of  this 
requirement  was  learning  the  vocabulary . One  student  providing  a mixed  review  listed  what  he/she  liked  most 
about  the  program  as  “Learning  new  vocabulary,”  and  what  he/she  liked  least  as  “Having  to  repeat  over  and 
over  the  same  vocabulary.”  Most  of  the  negative  comments  in  this  category  had  to  do  with  the  requirement  of 
typing  each  data  abnormality  name  multiple  times.  For  example,  one  student  wrote,  “Typing  in  lab  data  gets 
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repetitive  and  annoying,”  and  another  wrote,  ‘‘Sometimes  just  filling  out  the  names  of  all  the  abnormalities  was 
very  time  consuming.”  However,  it  seemed  clear  from  many  of  these  responses  that  it  was  the  repetitiveness, 
and  not  the  basic  requirement  of  generating  the  names  that  most  objected  to.  For  example,  the  same  student  who 
made  the  previous  comment  also  suggested,  “Perhaps  as  one  becomes  more  advanced,  lab  abnormalities  can  be 
pre -identified,  making  it  easier/quicker  to  complete  a case,”  suggesting  that  the  requirement  to  type 
abnormalities  is  an  acceptable  entry-level  requirement.  Other  students  seemed  to  object  to  the  requirement  of 
correct  spelling  at  all.  For  example,  one  student  wrote,  “I’d  have  a glossary  for  tests  and  abnormalities  just  for 
“pop  up”  spelling.  I get  mad  having  to  retype  for  spelling,”  and  another  wrote  that  the  software  would  be 
improved  by  “giving  a break  on  incorrect  spelling  by  1 or  2 letters.” 

5.  Requirement  of  completeness 

Ninety  seven  students  commented  on  the  aspects  of  the  software  that  require  the  learner  to  consider  all 
laboratory  data  when  constructing  a diagnostic  rationale,  or  to  classify  all  data  as  being  normal  or  not.  Of  those 
comments,  73  were  positive,  9 were  mixed,  and  15  were  primarily  negative.  Positive  comments  included, 
“couldn’t  ignore  any  abnormalities,”  “made  me  account  for  the  data  abnormalities,”  and  “It  made  me  analyze 
each  and  every  piece  of  data,  something  I probably  normally  would  not  have  done.”  One  of  the  students 
providing  a mixed  review  said,  “The  DP  made  me  get  more  lab  results  than  I probably  would  have  gotten 
otherwise,  but  I sometimes  found  myself  sifting  thru  them  in  a rote  manner.  Not  really  a DP  problem- more  my 
problem.”  Another  wrote  that  what  she/he  liked  best  about  the  DP  was  that  “It  makes  you  account  for  every 
abnormality  even  insignificant  ones.”,  while  what  she/he  liked  least  about  the  DP  was  that,  “It  makes  you 
account  for  every  abnormality  even  on  cases  where  the  solution  is  obvious.”  One  student  providing  primarily 
negative  comments  reported  liking  least,  “Having  to  account  for  every  extraneous,  insignificant  detail  that[’s] 
outside  the  norm.”  Another  reported  that  she/he  did  not  like,  “Having  to  account  for  morphology  results  that 
were  normal.  No  way  to  avoid  putting  them  in  the  path.” 

6.  Process  of  manipulating  data  in  the  diagnostic  path/format  of  diagnostic  path 

Eighty  nine  students  commented  on  the  process  of  manipulating  data  in  the  diagnostic  path.  Forty  six 
of  these  students  made  generally  positive  comments  about  this  process,  4 made  mixed  comments,  and  39  made 
primarily  negative  comments,  or  suggestions  for  improvement.  Many  of  the  positive  comments  had  to  do  with 
the  ease  of  manipulating  the  data.  For  example,  when  asked  what  they  liked  best  about  using  the  DP  one  student 
wrote,  “I  can  move  things  easier  than  erasing  them,”  another  wrote,  “neatness  and  ability  to  quickly  and  easily 
rearrange  diagnostic  paths”  and  a third  wrote,  “organizing  clinical  abnormalities  and  formulating  a diagnostic 
path.”  One  of  the  students  providing  a mixed  reviewobserved  as  a benefit  that  the  DP,  “helps  me  to  analyze 
data  more  efficiently,”  while  also  noting  as  something  she/he  didn’t  like  that  it  “takes  a long  time  to  construct  a 
path.”  Many  of  the  negative  comments  in  this  category  had  to  do  with  the  difficulty  inherent  in 
considering/presenting  all  the  relevant  data  at  once.  For  example,  one  student  wrote  that,  “Not  being  able  to  see 
the  entire  path  at  once  makes  it  difficult  to  organize  my  thought  further  along  the  path,”  and  another  observed 
that  “It  was  difficult  to  tell  what  signs/data  still  needed  to  be  placed  into  the  diagnostic  path.”  Other  students 
had  difficulty  with  specific  aspects  of  the  diagnostic  path  construction  process,  such  as  not  being  able  to  easily 
place  mechanisms  where  they  wanted  them  to  appear  in  the  diagnostic  path.  Finally,  some  students 
recommended  that  concept  map-style  diagrams  be  used  in  portraying  the  diagnostic  path,  as  opposed  to  the 
outline  format  currently  used. 

7.  Expert  feedback 

Eighty  three  students  commented  on  the  feedback  they  receive  regarding  their  rationale  in  the  form  of 
the  expert  diagnostic  path.  Sixty  eight  of  these  comments  were  strictly  positive;  15  were  mixed.  The  following 
are  illustrative  of  the  positive  comments : “I  can  compare  my  list  to  the  expert  list  right  away,”  “The 
professional  pathways  given  very  quickly,”  and  “I  liked  having  immediate  feedback;  I think  that  is  very 
beneficial  to  learning  something  new.”  The  fifteen  students  giving  mixed  comments  all  found  the  expert 
feedback  useful,  but  wanted  that  feedback  altered  or  expanded  in  some  way.  For  example,  one  student  wrote,  “I 
wish  the  expert  path  had  a few  more  notes  explaining  certain  things  instead  of  just  listing  them,”  and  another 
wrote,  “I  would  like  it  more  interactive  at  the  end  when  I compare  my  diagnosis  with  the  clinical  pathologist’s.” 
Several  students  also  indicated  wanting  access  to  the  expert  rationale  without  having  to  complete  the  case  first. 
None  of  the  comments  suggested  that  students  did  not  want  expert  feedback. 
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Discussion 

The  Likert  responses  provide  a valuable  perspective  for  interpreting  the  open-ended  responses.  Clearly, 
based  on  the  Likert  response  data,  most  of  the  negative  comments  or  suggestions  for  improvement  came  from 
students  who  overall  felt  the  DP  was  a useful  learning  tool.  This  is  supported  by  findings  from  comments  in  the 
first  category.  Overall,  positive  comments  outnumbered  mixed  comments  by  a ratio  of  1.6  to  1,  and  positive 
comments  outnumbered  negative  ones  approximately  12  to  1. 

The  greatest  number  of  positive  comments  fell  into  the  second  category,  which  we  have  called 
“ease/efficiency  of  thinking”  because  these  comments  tend  to  make  claims  regarding  the  DP’s  effect  of  helping 
to  organize  or  clarify  thoughts,  or  to  make  learning/thinking  easier.  These  comments  are  perhaps  the  most 
difficult  to  interpret  because  this  category  contains  the  most  vague  positive  statements.  However,  at  the  risk  of 
stating  the  obvious,  all  these  comments  must  refer  to  one  or  more  things  that  the  DP  does,  or  allows.  Also,  since 
the  same  broad  group  of  respondents  produced  comments  in  the  other  categories  we  have  provided,  it  seems 
likely  that  many  of  the  vague  positive  impressions,  if  the  students  could  have  been  prompted  for  specifics, 
would  have  fallen  into  one  of  the  broad  categories  3-7.  Similarly,  because  so  many  of  these  comments  seem  to 
center  on  the  organization  of  thinking,  it  seems  fair  to  say  that  these  indications  support  the  broad  claims  of 
cognitive  tools  enthusiasts  that  encouraging  students  to  create  representations  of  the  knowledge  relevant  to 
solving  a given  problem  will  enhance  their  ability  to  solve  that  (and  similar)  problems.  Furthermore,  the  fact 
that  the  enforced  organization  of  pathophysiologic  concepts  is  identified  by  the  students  as  helpful  to  learning, 
and  that  it  is  associated  with  greater  performance  on  related  problem  solving  tasks,  supports  the  idea  that 
improved  problem  solving  performance  is  related  to  practice  producing  robust  representations  of  the  problem’s 
sub-elements. 

As  seen  with  the  third  category,  one  hundred  and  fifteen  students  specifically  mentioned  finding  DP- 
use  convenient/easy.  The  main  idea  seems  to  be  that  many  students  consider  it  easier  to  deal  with/manipulate 
information  electronically  than  it  would  be  in  paper  form.  This  is  true  both  in  the  context  of  the  software  itself, 
where  concepts  can  be  identified  and  manipulated  without  the  necessity  of  erasing,  re-writing,  etc.,  as  well  as  in 
the  administrative  aspects  of  the  homework  process  (receiving  assignments,  partially  completing  and  saving 
assignments,  submitting  assignments,  etc.).  Clearly,  the  data  manipulation  aspects  of  this  preference  could  be 
theorized  to  have  a beneficial  impact  on  learning.  It  certainly  seems  reasonable  to  suggest  that  any  relief  an 
instructional  approach  may  pose  in  administrative  cognitive  overhead  will  allow  learners  to  focus  more 
attention  on  integrating/understanding  domain -specific  concepts. 

The  fourth  category,  the  one  requiring  students  to  enter  all  data  abnormality  names  manually  and 
spelled  correctly,  received  by  far  the  greatest  proportion  of  negative  comments.  In  fact,  the  number  of  students 
who  felt  this  policy  required  a change  outnumbered  those  who  made  positive  comments  about  it  by  a ratio  of 
1.7  to  1.  At  the  same  time,  among  the  forty  three  students  who  endorsed  this  feature,  as  well  as  among  many 
arguing  for  change,  there  was  strong  agreement  that  writing  data  abnormality  names  manually  resulted  in 
learning  the  vocabulary.  This  was  the  intended  outcome  of  this  feature.  It  will  be  our  goal  to  adjust  the  current 
requirements  to  the  point  that  student  learning  and  annoyance  are  optimized  against  each  other.  We  plan  to  do 
this  by  allowing  students  to  ‘‘pass  out”  of  the  spelling  requirement  by  spelling  data  abnormality  names  correctly 
the  first  few  times  they  encounter  them,  and  then  either  having  an  auto-fill  feature  or  a pull-down  list  become 
available  for  those  students. 

The  fifth  category  referred  to  various  gating  behaviors  of  the  DP  that  require  students  to  consider  all 
data  that  technically  fall  outside  of  the  reference  ranges  for  every  test  in  any  given  case  (even  if  such  data  turns 
out  to  be  clinically  insignificant).  This  feature  was  viewed  as  beneficial  for  learning  by  many  students,  with 
positive  comments  outnumbering  negative  comments  or  suggestions  for  improvement  by  a ratio  of  4.9  to  1.  In 
essence,  many  students  felt  that  they  considered  more  information  than  they  would  have  considered  had  they 
not  used  the  DP,  and  that  doing  so  resulted  in  superior  understanding  of  the  underlying  pathology  and 
physiology.  While  a reasonable  fear  might  be  that  this  aspect  of  the  software  would  serve  as  a crutch  rather  than 
a scaffold,  with  students  returning  to  “ignoring”  behavior  with  the  DP  withdrawn,  that  does  not  seem  to  have 
been  the  case,  given  the  students’  improvement  on  case-based  exams  that  were  entirely  paper  based. 

The  sixth  category  involves  the  process  of  manipulating  data  in  the  diagnostic  path.  Recall  that  this  is  a 
process  of  dragging  and  dropping  concepts  or  clusters  of  concepts  in  an  outline  format,  where  it  is  understood 
that  items  above  and  to  the  left  cause  items  below  and  to  the  right,  or  items  below  and  to  the  right  are  supportive 
of  items  above  and  to  the  left.  Students  having  only  positive  things  to  say  about  this  process  slightly 
outnumbered  those  having  primarily  criticisms  or  suggestions  for  improvement  (1.2  to  1).  Clearly,  this  process 
worked  well  for  many  students,  but  also  was  difficult  for  a significant  number  of  others.  While  some  other 
formalism  for  representing  the  learners’  knowledge  might  be  less  problematic  for  more  students,  there  are 
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reasons  to  suspect  that  this  process  will  be  difficult  regardless  of  the  formalism  used.  First,  the  information- 
synthesis  task  is  very  difficult,  regardless  of  the  mechanisms  used  to  assist  the  learner  in  accomplishing  it.  In 
this  case,  students  had  to  represent  causal  and/or  supportive  knowledge  involving  dozens  of  discreet  concepts  in 
most  cases,  and  more  than  a hundred  discreet  concepts  for  complicated  cases.  Given  the  known  limitations  of 
human  cognition,  this  process  is  inevitably  difficult  regardless  of  the  information-presentation  formalism  that  is 
employed.  Second,  current  commonly  available  electronic  display  technology  doesn’t  permit  for  the  spatially 
meaningful  representation  of  large  numbers  of  concepts  simultaneously.  This  number  can  be  very  low  for 
students  with  older  computers  that  only  display  at  a resolution  of  640X480  or  800X600  pixels.  Therefore,  some 
students  would  not  be  able  to  represent  as  many  concepts  using  the  DP  as  they  might  be  able  to  do  using  other 
traditional  mechanisms  of  data  representation,  such  as  a blackboard,  or  a large  piece  of  paper.  In  summary,  our 
future  research/development  will  explore  providing  optional  ways  of  representing  the  diagnostic  rationale , such 
as  concept  maps,  with  the  hope  of  maximizing  clarity  and  spatial  efficiency,  and  accommodating  individual 
learner  preferences.  At  the  same  time,  however,  we  suspect  that  the  complexity  of  the  task  coupled  with  display 
limitations  will  continue  to  be  problematic  for  learners,  regardless  of  the  formalism  used  to  display  the  data. 

The  data  presented  in  the  seventh  category  show  significant  support  for  the  DP’s  feedback  feature 
whereby  students  receive  feedback  regarding  their  rationale  in  the  form  of  an  “expert  rationale’’  created  by  an 
expert  clinical  pathologist  completing  the  identical  case  using  the  DP.  All  respondents  who  mentioned  this 
feature  felt  that  this  feature  was  valuable,  with  some  requests  that  it  be  expanded  in  some  way.  This  should 
come  as  no  surprise  to  most  readers  - immediate  and  meaningful  feedback  is  prescribed  by  many  common 
instructional  approaches,  and  these  data  support  the  general  idea  that  feedback  is  a good  idea.  Again,  one 
particular  strength  of  this  approach  is  that  the  expert  and  the  student  both  used  the  same  process  to  construct  the 
diagnostic  rationale,  making  it  possible  to  attribute  differences  between  student  and  expert  rationale  to  different 
understandings  of  the  underlying  concepts,  rather  than  to  differences  in  the  representation  of  the  problem. 

Implications  for  the  Design  of  Software  Tools 

One  of  the  complexities  of  attempting  to  meaningfully  synthesize  open-ended  responses  is  that 
structure  and  order  have  not  been  pre -imposed  on  the  resulting  data,  as  commonly  occurs  with  more  traditional 
empirical  approaches  to  research.  The  results  can  frequently  represent  a hodgepodge  of  ideas.  In  this  case, 
however,  we  feel  that  several  strong  coherent  concepts  emerged  that  can  be  useful  to  the  designer  of  software 
learning  tools.  These  ideas  should  be  evident  in  the  discussion  section.  Here  we  briefly  review  them.  1.  Our 
findings  appear  to  support  the  general  cognitive  tool  concept  - that  requiring  learners  to  create  detailed 
representations  of  the  relationships  between  the  concepts  required  to  solve  a given  problem  will  promote 
understanding  and  problem  solving  ability  in  that  domain.  2.  Meaningful  feedback  is  important.  In  our  context, 
producing  and  presenting  feedback  in  the  same  manner  used  by  students  to  create  their  rationale  appears  to  have 
been  beneficial.  3.  Students  saw  benefit  in  the  DP’s  effect  on  the  organization  of  their  thoughts.  This  was 
accomplished  in  two  ways:  first,  through  gating  the  tasks  relevant  to  solving  the  problem  (data  identification 
first,  followed  by  data  synthesis),  and  second,  through  use  of  the  outline -based  diagnostic  path  formalism.  4. 

The  ease/convenience  of  electronically  manipulating  problem  elements,  and  managing  the  larger  learning 
process  was  appreciated  by  many  students,  and  may  have  decreased  the  “data  management’’  cognitive 
requirements  for  students. 


Limitations  and  Future  Directions 

This  study  is  part  of  a larger  research  project.  These  Likert -based  and  open-ended  response  data  are 
intended  to  explore  the  specific  aspects  of  the  DP  that  produced  the  learning  gains  that  were  seen.  While  these 
findings  seem  compelling,  the  fact  that  they  are  based  on  the  subjective  analysis  of  open-ended  response  data 
suggests  further  studies  to  explore  these  principles  empirically.  For  example,  a version  of  the  DP  could  be 
created  that  relaxes  gating  requirements.  Second,  the  problem-solving  tasks  in  which  the  students  engaged 
constitute  complete  and  authentic  diagnostic  problem  solving  tasks  from  the  perspective  of  clinical  pathology. 
Often,  all  the  data  a clinical  pathologist  has  to  consider  are  laboratory  data,  signalment,  and  a brief  history. 
However,  this  process  is  not  representative  of  the  broader  clinical  problem  solving  process,  which  involves 
additional  data  and  data  collection,  as  well  as  the  handling  of  therapy.  In  the  future,  we  must  further  explore 
outcomes  of  DP-use  using  broader  measures  of  problem  solving,  and  including  clinical  problem  solving 
performance. 
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Effects  of  Worked  Examples  and  Conventional  Problem-Solving 
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Abstract 

Thirty  six  senior  students  in  chemical  engineering  were  randomly  assigned  to  three  treatment 
groups  in  an  experimental  study  that  examined  the  impact  of  different  instructional  strategies  for 
troubleshooting  malfunctions  in  a computer-based  simulation  of  a chemical  processing  plant.  In  two 
groups,  different  types  of  worked  examples,  process-oriented  and  product-oriented,  were  given  to 
participants  as  instructional  strategies  for  troubleshooting  four  plant  malfunctions.  The  third  group 
was  given  a conventional  problem  solving  strategy  for  the  same  four  problems.  The  results  of 
participants  ’ performance  on  solving  a set  of  eight  near-transfer  problems  indicated  no  significant 
transfer  differences  among  the  treatments.  Neither  did  afar  transfer  task  result  in  any  significant 
differences.  The  findings  of  the  current  study  supported  the  notion  of  the  “expertise  reversal  effect” 

( Kalyuga , Ayres,  Chandler,  & Sweller,  2003),  which  argues  that  presenting  new  information  to 
learners  with  pre-existing  schemata  in  a given  domain  does  not  improve  transfer  and  may  induce 
extraneous  cognitive  load.  Given  the  prior  knowledge  of  the  participants,  these  findings  were  also 
consistent  with  Sweller’s  (2004)  thesis  on  the  “central  executive  function”  and  his  description  of  the 
“redundancy  effect.  ” 

In  his  analogy  between  evolution  by  natural  selection  and  human  cognitive  architecture,  Sweller 
(2004)  lists  the  assumptions  of  cognitive  load  theory.  The  theory  assumes  that  the  purpose  of  instruction  is  to 
build  knowledge  by  making  small  incremental  changes  in  long-term  memory.  He  argued  that,  similar  to  the  way 
a potentially  injurious  drastic  alteration  in  the  human  genome  is  usually  prevented  by  the  process  of  natural 
selection  in  species,  a sweeping  change  in  an  individual’s  long-term  memory  is  prevented  by  the  severe 
limitation  of  working  memory  when  assimilating  unfamiliar  information.  Such  assimilation  when  no  schema 
exists  for  organizing  new  information  is  performed  by  searching  and  testing  the  fit  of  random  combinations  of 
elements  in  the  new  material  against  premises  derived  from  established  assumptions  retrieved  from  long-term 
memory.  The  demand  on  working  memory  is  raised  exponentially  as  the  number  of  unfamiliar  interacting 
elements  of  information  is  increased.  The  random  search  is  essential  to  the  human  cognitive  architecture  when 
learners  face  completely  unfamiliar  information  and  a central  executive  is  absent.  However,  when  facing 
familiar  information  as  opposed  to  unfamiliar  information,  a highly  effective  central  executive  function 
becomes  available.  As  opposed  to  the  human  genome,  this  function  is  not  a general  biological  structure,  but  a 
specific  learned  structure  retrieved  from  long-term  memory.  In  other  words,  cognitive  processes  conditioned  by 
domain -specific  knowledge  act  as  the  central  executive  when  sufficient  elements  of  instructional  material  are 
familiar.  On  the  other  hand,  when  the  learner  lacks  a central  executive  because  the  information  is  unfamiliar,  the 
use  of  worked  examples  in  the  design  of  instructional  material  can  provide  a surrogate  central  executive  that 
constrains  the  problem  space  and  the  number  of  interacting  elements  to  be  randomly  searched. 

Based  on  Sweller’s  (2004)  argument,  when  instruction  is  properly  designed,  effective  changes  to  long- 
term memory  structures  are  orderly  and  occur  in  small  increments.  Cognitive  load  theory  provides  guidelines 
for  designing  instruction.  Investigating  and  expanding  upon  these  guidelines,  researchers  have  identified 
instructional  strategies  that  can  facilitate  incremental  changes  in  long-term  memory.  The  use  of  worked 
examples  is  one  of  those  instructional  strategies  that  promotes  efficient  and  effective  learning  by  reducing 
extraneous  cognitive  load  through  the  introduction  of  schemata  and  by  acting  as  the  “instructional  central 
executive’’  (p.  21)  and  accommodating  the  limited  capacity  of  the  working  memory  of  novices. 

When  teaching  complex  cognitive  skills  to  novices,  the  instructional  strategy  of  presenting  a set  of 
worked  examples  for  learners  to  study  has  been  repeatedly  found  more  effective  than  the  conventional  problem 
solving  strategy  in  which  they  are  provided  problems  to  solve  immediately  after  presentation  of  information  in 
the  domain  (e.g..  Cooper  & Sweller,  1987;  Paas,  1992;  Paas  & van  Merrienboer,  1994a;  Sweller  & Cooper, 

1985;  for  a review,  see  Atkinson,  Derry,  Renkl,  & Wortham,  2000).  Two  types  of  worked  examples,  process- 
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oriented  and  product -oriented,  have  been  distinguished  with  regard  to  the  cognitive  load  those  worked  example 
strategies  impose  on  learners  and  the  instructional  efficiency  of  those  types  (Van  Gog,  Paas,  & Van 
Merrienboer,  2004).  Specifically,  Van  Gog,  et  al.  proposed  that  in  the  initial  instruction  of  novices,  the  process- 
oriented  worked  example  strategy,  which  explains  not  only  how  to  solve  a given  problem  but  why  the  operations 
are  employed,  would  result  in  greater  problem  solving  performance  and  transfer.  In  contrast,  the  product- 
oriented  worked  examples  strategy  that  just  describes  the  procedures  involved  in  solving  a problem  would  be 
more  effective  only  after  a learner  has  constructed  relevant  schemata. 

Based  on  Van  Gog,  et  al.’s  (2004)  contentions,  process -oriented  worked  examp  les  should  be  more 
effective  when  used  with  novices  who  have  established  relevant  schemata  prior  to  instruction.  They  would 
therefore  benefit  from  the  knowledge  provided  by  why  problem  solving  principles.  To  our  knowledge,  no 
empirical  studies  have  been  conducted  to  examine  both  of  these  strategies  at  once  for  their  impact  on  the 
instructional  outcomes.  The  current  study  investigated  the  effects  of  these  two  types  of  worked  examples  with  a 
control  condition  that  employed  conventional  problem  solving.  It  reports  measured  effects  on  performance  of 
acquired  skills  in  troubleshooting. 


Method 

Participants 

Thirty-six  senior  engineering  students  enrolled  in  a Chemical  Engineering  Design  course  offered  by 
the  Florida  A&M  University  - Florida  State  University  College  of  Engineering  participated  in  the  study  as  part 
of  a required  class  assignment.  They  engaged  in  this  activity  as  a required  assignment  in  their  final  semester  of 
the  bachelor’s  degree  program.  Twenty-one  of  the  participants  were  male  and  15  were  female.  All  except  one 
were  Chemical  Engineering  majors  and  had  taken  courses  that  introduced  concepts  of  distillation. 

Procedure 

The  participants  engaged  in  instruction  about  a water-alcohol  distillation  plant  as  a simulation 
specifically  designed  (De  Croock  & Betlem,  1999)  for  experiments  in  the  area  of  complex  cognitive  skills  (see 
Figure  1).  The  initial  instruction  on  how  to  operate  the  simulation  was  the  same  for  each  participant.  In  the 
following  treatment,  three  instructional  strategies  were  employed:  (1)  process-oriented  worked  examples,  (2) 
product-oriented  worked  examples,  and  (3)  conventional  problem  solving.  Each  of  the  three  treatment  groups 
encountered  the  same  four  faults  in  the  plant.  For  even  distribution  of  participants  with  varying  degrees  of  prior 
knowledge  in  distillation,  the  subjects  were  divided  into  two  categories  of  high  and  low  according  to  their 
scores  in  a recent  course  that  taught  them  distillation.  Equal  numbers  of  participants  in  high  and  low  categories 
were  then  randomly  assigned  to  the  three  treatment  groups  and  were  given,  process-oriented  worked  examples 
(PC),  product-oriented  worked  examples  (PD),  and  conventional  problem  solving  (PB). 

Following  the  instruction,  as  a near-transfer  task,  all  36  participants  diagnosed  eight  malfunctions  they 
had  not  previously  encountered,  for  which  they  were  limited  to  12  minutes  for  each  malfunction.  Participants 
were  told  to  “make  as  few  incorrect  diagnoses  as  possible  and  diagnose  the  malfunction  as  quickly  as  possible.” 
A far  transfer  task  designed  in  CHEMCAD  measured  the  number  of  trials  for  participants  to  solve  a problem 
conceptually  related  to  the  near  transfer  task.  CHEMCAD  is  a computer  simulation  program  used  by  chemical 
engineers. 

Performance  Measures 

Three  measures  of  performance  were  used  to  assess  learners’  performance.  Those  three  measures  were 
(1)  the  total  number  of  correct  diagnoses  within  the  12  minute  limit,  (2)  the  number  of  incorrect  diagnoses 
participants  reported,  and  (3)  the  time  required  to  diagnose  a malfunction  correctly. 

Mental  Effort 

The  9-point  Mental  Effort  Scale  (Paas  & van  Merrrienboer,  1994b)  measured  the  subjects’  perceived 
mental  effort  invested  in  performing  the  tasks.  At  the  high  end  of  the  scale,  9 was  associated  with  the  response 
“very,  very  high  mental  effort”  and  at  the  low  end  of  the  scale,  1 was  associated  with  the  response  “very,  very 
low  mental  effort.”  The  scale  was  administered  immediately  following  each  correct  diagnosis  and  repair  to 
provide  a subjective  rating  of  the  variable  “cognitive  load.” 

Results 

With  alpha  set  at  .05,  an  ANOVA  revealed  no  significant  differences  among  the  treatment  conditions 
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on  any  of  the  dependent  measures  of  performance.  Table  1 displays  the  mean  performance  scores  and  standard 
deviations  for  treatment  groups,  summed  across  the  eight  problems,  along  with  reports  of  perceived  mental 
effort.  The  far  transfer  performance  measured  in  number  of  trials  to  solve  the  problem  are  also  reported  in  Table 
1. 


Table  1 Means  and  standard  deviations  of  dependent  measures  summed  across  eight  problems 


Process- 

oriented 

worked 

examples 

(PC) 

Product- 

oriented 

worked 

examples 

(PD) 

Conventional 

problem 

solving 

(PB) 

M 

SD 

M 

SD 

M 

SD 

Correct  diagnoses 

6.91 

1.00 

6.75 

0.97 

i.\i 

0.83 

Incorrect  diagnoses 

27.3 

3 

13.0 

9 

32.3 

3 

12.3 

5 

25.7 

5 

7.88 

Time  to  correct 
diagnoses  (in 
seconds) 

10®  402 

6 

1097 

320 

103 

4 

357 

Perceived  mental 
effort 

44.3 

3 

11.2 

2 

46*j  8.86 

40.4 

1 

6.88 

Far  transfer 
performance 

14'°  5.87 

15'2  7.25 

17.5 

8 

12.8 

5 

Further  analysis  of  the  data  revealed  an  unexpected  difference  between  male  and  female  participants 
on  two  of  the  performance  measures.  Females  had  a significantly  (p  = 0.02)  higher  number  of  incorrect 
diagnoses  (M  = 33.60,  SD  = 11.35)  than  males  ( M = 24.81,  SD  = 10.12).  They  also  took  less  time,  calculated  in 
seconds  (M  = 870.93,  SD  = 227.52),  than  their  male  counterparts  (M  = 1215.62,  SD  = 332.95)  to  correctly 
diagnose  the  malfunctions  (p  = 0.002). 


Discussion 

In  the  context  of  Sweller’s  (2004)  analogy  of  human  cognitive  architecture  to  evolution  by  natural 
selection,  worked  examples  can  provide  an  instructional  central  executive  when  none  exists  in  the  domain 
represented  by  the  learning  tasks.  Worked  examples  are  found  to  be  effective  (Cooper  & Sweller,  1987;  Paas, 
1992;  Paas  & van  Merrienboer,  1994a;  Sweller  & Cooper,  1985)  because  they  cause  incremental  changes  in 
long-term  memory  through  acquisition  of  the  new  information  they  present,  thus  building  new  schemata. 
However,  if  a learner’s  long-term  memory  contains  pre-existing  schemata  for  solving  problems  in  the  domain, 
the  instruction  would  not  be  effective  and  could  even  impose  extraneous  cognitive  load. 

In  this  context,  the  process-oriented  worked  examples  were  expected  to  contribute  to  the  participants’ 
performance  more  than  the  other  strategies  by  providing  the  elements  for  building  new  schemata.  We  argue  that 
the  ineffectiveness  of  the  worked  examples  demonstrated  by  the  results  of  this  study  was  due  to  high  level  of 
the  participants’  prior  knowledge  and  the  existence  of  schemata  for  solving  these  types  of  problems.  It  seems 
likely  that  the  participants,  who  were  already  familiar  with  the  principles  of  distillation,  gained  little  from  the 
principled  reasoning  presented  in  the  worked  examples.  Indeed,  they  might  have  experienced  the  “expertise- 
reversal  effect”  (Kalyuga,  Ayers,  Chandler,  & Sweller,  2003;  Kalyuga,  Chandler,  Tuovinen,  & Sweller,  2001), 
in  which  instruction  designed  to  facilitate  construction  of  schemata  conflicts  with  learners’  existing  schemata 
and  thus  inhibits  their  understanding. 

According  to  the  data  presented  in  Table  1,  all  three  treatment  groups  correctly  solved  an  average  of 
seven  of  the  eight  problems  presented  to  them.  We  attribute  this  rather  high  performance  to  the  participants’ 
prior  knowledge  which  rendered  the  strategies  practically  ineffective  for  these  participants.  Further  support  for 
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this  argument  is  apparent  in  the  participants’  perceived  mental  effort  also  presented  in  Table  1.  According  to 
this  information,  the  mean  mental  effort  for  the  three  groups  across  the  eight  problems  they  solved  was  near  the 
midpoint  of  the  mental  effort  scale.  They  reported  an  average  mental  effort  of  40,  44,  and  47  out  of  a possible 
score  of  72  across  the  eight  tasks.  These  are  relatively  low  mental  effort  scores  for  solving  complex  problems 
and  indicate  that  participants  had  little  difficulty.  This  argument  is  further  substantiated  by  participants’ 
performance  on  the  far  transfer  task.  All  of  the  participants  correctly  solved  the  CHEMCAD  problem  with  an 
average  number  of  trials  (M  = 15.86)  much  lower  than  expected. 

We  speculate  that  novice  participants,  given  the  same  instructional  treatments  and  experimental 
conditions,  would  respond  differently  to  the  different  types  of  worked  examples.  Using  process-oriented  worked 
examples,  novices  would  be  expected  to  perform  better  than  those  using  the  other  strategies.  However,  they 
would  be  expected  to  invest  much  higher  mental  effort  in  the  problem  solving  process.  In  summary,  the 
conventional  problem  solving  exercises  were  probably  more  suitable  for  these  participants  and  the  use  of 
worked  examples  made  little  difference  in  their  performance.  These  findings  further  substantiate  our  argument 
that  the  participants’  prior  knowledge  accounted  for  the  worked  examples  not  being  significantly  different  from 
the  conventional  problem  solving  strategy  in  their  contribution  to  the  participants’  performance. 

Further  analysis  of  the  performance  measures  revealed  two  unexpected  results  among  the  participants. 
Females  made  more  incorrect  diagnoses  than  males  and  took  less  time  to  diagnose  malfunctions.  We  attribute 
this  difference  to  the  instruction  given  at  the  beginning  of  the  performance  phase  of  the  experiment.  We  told  the 
participants  to  “diagnose  the  malfunction  as  quickly  as  possible’’  and  “make  as  few  incorrect  diagnoses  as 
possible.’’  Apparently  males  and  females  responded  differently  to  these  instructions.  Based  on  the  results  of  the 
study,  each  group  favored  only  one  portion  of  the  instructions.  An  explanation  for  the  reasons  for  these 
differences  between  the  diagnostic  behaviors  of  males  and  females  could  be  the  subject  of  further  research. 

Future  investigation  can  also  focus  on  the  effectiveness  of  the  strategies  used  in  this  study  by  involving 
novice  participants  in  the  same  experimental  conditions.  Worked  examples  - process-oriented  and  product- 
oriented  - along  with  the  conventional  problem  solving  strategy  may  demonstrate  a significant  difference  in 
novices’  performance.  The  lack  of  a central  executive  in  novices’  long-term  memory  structures  - or  schemata  - 
should  reveal  the  different  effects  of  the  instructional  strategies.  We  suggest  that  replicating  the  experiment  with 
novice  learners  would  provide  a set  of  data  by  which  one  can  compare  the  results  with  those  of  this  study. 
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Abstract 

The  purpose  of  this  presentation  is  to  continue  the  dialogue  on  the  professional  status  of  and  the 
specialization  within  the  instructional  design  and  technology  (IDT)  field.  This  presentation  highlights  the 
similarities  and  differences  among  the  various  classification  systems,  which  are  used  for  ranking  occupational 
fields,  and  discusses  the  most  appropriate  system  for  categorizing  the  professional  status  for  IDT.  A second 
purpose  is  to  discuss  the  delineation  of  specialization  within  the  IDT  field. 

Introduction 

The  IDT  literature  is  replete  with  information  on  the  field's  historical  roots  and  its  current  state 
(Dempsey  & Van  Eck,  2002;  Ely,  1998;  Reiser,  2002;  Saettler,  1990;  Torkelson,  1998)  since  the  inception  of 
AECT  in  1977.  Although  it  is  often  discussed  as  a profession  (Bratton,  1981;  Davidson,  1985,  1987),  the  IDT 
field  does  not  have  a clearly  r defined  professional  status  based  on  any  formal  classification  system  or  a 
comparison  of  occupations,  which  are  considered  full  status  professions. 

Depending  on  the  type  of  categorization  or  approach  used,  any  field  or  occupation  varies  in  its  ranking, 
or  status.  Hence,  determining  the  professional  status  of  the  IDT  field  has  remained  elusive  due,  in  part,  to  no 
single  authoritative  classification  system  appearing  to  fit  well  with  it.  Consequently,  the  fundamental  question 
remains,  is  IDT  a profession?  If  so,  how  is  the  professional  status  characterized?  In  order  to  determine  whether 
IDT  is  a profession  several  considerations  must  be  made.  First  and  foremost  is  to  define  the  term,  profession. 

What  Do  We  Mean  by  Profession? 

The  term,  profession,  is  often  bandied  about  by  practitioners  and  is  used  in  common  language.  For 
instance,  the  descriptor  professional  can  be  seen  on  delivery  vehicles,  in  advertisements,  and  self-proclaimed  in 
proprietary  literature.  Often,  “profession”  is  simply  used  to  refer  to  the  work  that  an  individual  does  for  a living. 
Furthermore,  an  individual  is  called  a “professional”  to  convey  a level  of  dignity  and  prestige.  Additionally,  the 
professional  craftsman,  such  as  an  electrician  or  plumber  is  differentiated  from  the  amateur  by  the  term  of 
profession,  again,  with  the  expectation  that  the  professional  has  experience,  competence,  and  is  often  licensed 
(Pavalko,  1971). 

Even  a dictionary  derived  definition,  which  states  that  a profession  is  "an  occupation  requiring 
considerable  training  and  specialized  study:  medicine,  law,  engineering ” (The  American  Heritage  Dictionary, 
1996,  p.  1446)  does  not  provide  a clear  conception  of  what  it  is  to  be  a profession.  Hence,  there  have  been 
attempts  within  the  sociology  of  work  to  define  profession,  but  definitions  have  remained  ambiguous  at  best 
(Freidson,  1986;  Pavalko,  1971).  For  instance,  Cogan  (cited  in  Vollmer  & Mills  1966,  p.  vii)  offers  this 
definition: 

"A  profession  is  a vocation  whose  practice  is  founded  upon  an  understanding  of  the 
theoretical  structure  of  some  department  of  learning  or  science,  and  upon  the  abilities 
accompanying  such  understanding.  This  understanding  and  these  abilities  are  applied  to  the 
vital  practical  affairs  of  man.  The  practices  of  the  profession  are  modified  by  knowledge  of  a 
generalized  nature  and  by  the  accumulated  wisdom  and  experience  of  mankind,  which  serve 
to  correct  the  errors  of  specialism.  The  profession,  serving  the  vital  needs  of  man,  considers 
its  first  ethical  imperative  to  be  altruistic  service  to  the  client." 

Cogan's  definition  focuses  on  expert  knowledge  or  abilities  of  an  occupation  and  those  individuals 
associated  with  it.  His  definition  embodies  the  concepts  of  the  linkage  of  theory  to  practice,  how 
empirical  research  informs  theory  and  practice,  the  client-service  focus,  and  the  importance  of  the 
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service  to  society.  Others  would  identify  specialized  knowledge  and  abilities  as  part  of  a profession 
(Friedson,  1994;  Metzger,  1976). 

Basing  her  definitions  on  a review  of  literature,  Davidson  (1985,  1987)  distinguishes  profession  from 
occupation  by  stating  that  a "profession  is  an  acknowledged  vocation  requiring  extensive  education  in  science 
or  liberal  arts;  a calling"  whereas  an  occupation  is  "considered  to  be  a means  of  fillings  one's  time  with  regular 
employment.”  This  definition  uses  the  conceptualization  of  comparing  work  along  a broad  stratum  according  to 
Freidson  (1994),  which  is  similar  to  Flexner's  system  (as  cited  in  Metzger). 

More  simply  put,  Abbott  (1988,  p.  8)  defines  profession  as  ". . . exclusive  occupational  groups 
applying  somewhat  abstract  knowledge  to  particular  cases”  and  have  jurisdiction  over  the  body  of  specialized 
knowledge,  skill,  and  work  activity.  Abbott's  definition  again  refers  to  specialized  knowledge  related  to  work  or 
occupation  along  with  some  governance  over  that  knowledge,  but  does  not  necessarily  set  up  a comparison 
among  work  or  occupations.  The  key  to  Abbott’s  definition  is  the  occupational  group’s  exclusive  jurisdiction 
over  the  special  body  of  knowledge  and  work  as  a result  of  competition  and  negotiation  with  similar  groups  of 
challengers. 

Furthermore,  Abbott's  definition  provides  a more  external  system  orientation  toward  professions. 
Another  author  less  concerned  with  individual  traits,  Freidson  (1994,  p.  10),  refers  to  a profession  as  “an 
occupation  that  controls  its  own  work,  organized  by  a special  set  of  institutions  sustained  in  part  by  a particular 
ideology  of  expertise  and  service.” 

Although  there  is  no  definitive  statement  as  to  what  a profession  is,  examining  the  various  definitions 
may  facilitate  a better  understanding  of  the  term  and  its  varying  use.  One  particular  use  of  the  definitions  has 
been  to  classify  work  or  occupations  as  professional  or  having  professional  status.  For  instance,  a definition, 
found  in  the  Flexner  report,  is  based  on  a profession  having  a number  of  attributes  and,  according  to  Metzger 
(1976),  formed  a checklist  approach  to  the  definition  of  profession  (p.  43).  Furthermore,  in  order  for  an 
occupation  to  qualify  as  a profession  under  Flexner's  approach,  all  attributes  had  to  be  exhibited  by  the 
occupation.  In  turn,  Flexner's  definition  is  considered  as  one  of  the  classification  systems  for  determining  the 
professional  status  or  professionalization  of  work  or  an  occupation;  there  are  other  systems. 

Caplow  (cited  in  Vollmer  & Mills,  1966,  p.  20-21)  contends  that  there  is  a definite  sequence  of 
professionalization,  the  process  by  which  an  occupation  becomes  a profession.  Deduced  from  his  case  studies, 
each  occupation  that  has  achieved  professional  status  has  gone  through  a particular  sequence  of  events.  This 
sequence  includes:  establishing  of  a professional  organization,  changing  of  the  name  of  the  occupation  by  which 
a monopoly  of  work  can  be  secured,  developing  a code  of  ethics,  and  concurrently  both  lobbying  for  political 
power  to  maintain  the  new  work  monopoly  and  establishing  training  or  education  facilities.  One  could  question 
as  to  whether  we  should  look  at  these  as  milestones  to  be  addressed  or  merely  a report  of  what  has  happened  in 
the  past,  but  what  may  not  be  a viable  course  of  action  in  today’s  political-economic  ecology. 

What  are  the  Models  of  Professions? 

To  understand  the  application  of  the  term,  profession,  to  any  occupation,  we  may  ask  ourselves,  who  is 
it  that  ascribes  this  term? 

Economic  Skill-based  Model 

The  Department  of  Labor’s  skill-based  Standard  Occupational  Classification  (SOC)  system  (2000)  is 
an  economic  model  and  provides  one  method  of  classification  of  all  occupations.  This  system  examines  the 
nature  of  the  work  activity  in  combination  with  the  skills,  formal  preparation,  and  credentials  required  for  that 
work  activity.  This  classification  system  is  used  to  facilitate  statistical  analysis  for  informational,  policy,  and 
program  purposes.  Eight  hundred  occupations  are  included  under  the  23  major  groupings.  One  major  group  is 
titled  “Professional  and  related  occupations.”  Under  antecedent,  census-based  classification  schemes  this  group 
was  known  as  “professional  and  semiprofessional  workers  (1940)”  and  later  “professional,  technical,  and 
kindred”  occupations  (1950).  Though  the  SOC  system  does  not  specifically  define  the  term  “profession,”  its 
earlier  manifestations  stated  that  this  group  “performs  advisory,  administrative,  or  research  work  which  is  based 
upon  the  established  principles  of  a profession  or  science. ..and  requires... training  equivalent  to  that  represented 
by  graduation  for  a college  or  university. ..or  extensive  practical  experience”  (p.  1 16).  This  major, 
heterogeneous  grouping  includes  such  diverse  fields  as  medicine,  law,  the  clergy,  engineering,  architecture, 
computer  and  math  occupations,  scientists  of  all  stripes,  education  and  training,  the  media,  and  entertainment. 
We  now  turn  to  the  sociological  literature  of  the  study  of  professions  to  gain  some  understanding  of  how  the 
term  is  applied  and  what  constitutes  a profession. 
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Models  from  Sociology 

When  considering  the  various  models  of  professionalism  developed  over  the  last  century  and  a half,  it 
is  critically  important  to  keep  in  mind  the  social,  political,  and  economic  history  of  this  period  (Larson,  1990). 
America  grew  from  an  agrarian  society  to  an  industrial,  post-industrial,  and  informational  society.  Some 
occupations  were  created  and  others  were  superceded  by  technology,  the  growth  of  scientific  knowledge,  and 
changes  in  market  demand.  According  to  Kerr  (1983),  there  are  two  broad  classification  systems  alternatively 
called  “trait  models”  and  “power  models.” 

Trait  Models 

Metzger  (1976)  noted  that  early  in  the  twentieth  century,  Flexner  evaluated  the  fledgling  medical 
profession  and  suggested  that  certain  characteristics  or  traits  were  required  for  medicine  or  any  other 
occupation,  to  become  considered  a profession.  Metzger  suggested  that  Flexner's  model  required  that  an 
occupation  must  possess  seven  traits  in  order  to  be  a profession  and  it  formed  the  bases  of  trait -based  models. 
Metzger  further  noted  that  Flexner  had  a profound  effect  on  social  scientists  studying  the  emergence  of 
professionalism  in  society.  As  the  number  of  proclaimed  professions  grew,  so  did  the  field  of  sociology  in  the 
study  of  professions.  Consensus  was  limited  and  the  list  of  requirements  grew  or  shrank  as  dictated  by  the 
analyst’s  perspective  as  new  occupations  were  examined.  Several  of  Flexner's  criteria  were  elaborated  or  re- 
articulated  adding  the  requirement  for  being  a life  -long,  full-time  occupation,  a calling  with  a service 
orientation,  with  limits  to  entry,  autonomy,  employment  of  discretionary  practices,  and  codes  of  ethics. 

Millerson  (1964)  conducted  a meta-analysis  of  twenty-one  lists  of  professional  traits  and  concluded 
that  the  essential  elements  of  a profession  were  the  six  most  frequently  listed.  They  are  as  follows:  (a)  a 
profession  involves  a skill  based  on  theoretical  knowledge,  (b)  the  skill  requires  training  and  education,  (c)  the 
professional  must  demonstrate  competence  by  passing  a test,  (d)  integrity  is  maintained  by  adherence  to  a code 
of  conduct,  (e)  the  service  is  for  the  public  good,  and  (f)  the  profession  is  organized  (p.  4). 

Pavalko’s  (1971)  model  of  the  “occupation-profession  continuum”  is  representative  of  contemporary 
trait-based  models  and  has  eight  dimensions.  These  dimensions  include:  (1)  theory  or  intellectual  technique, 

(2)  relevance  to  basic  social  values,  (3)  training  period,  (4)  motivation,  (5)  autonomy,  (6)  sense  of  commitment, 
(7)  sense  of  community,  and  (8)  code  of  ethics.  Pavalko,  among  others,  argues  that  occupations  lie  somewhere 
along  a continuum  in  the  professionalization  process  and  that  this  process  is  neither  unilinear  nor  static  (Parelius 
& Parelius,  1987).  Similarly,  Moore  (1970),  using  a “scale”  perspective,  considers  some  criteria  of  a higher 
order  than  other.  He  also  recognizes  that  none  of  the  profession’s/quasi-,  near-profession’s  practitioners  are 
homogeneous  and  their  individual  position  lies  on  points  of  the  scale  perhaps  differently  than  that  of  the  field 
overall. 

Additionally,  Pavalko  introduces  the  idea-“marginalization”  where  some  occupations  may  hold  a 
position  toward  the  professional  end  of  the  continuum  in  several  of  his  model’s  dimensions,  but  are  on  the 
opposite  end  in  others.  For  example,  Pavalko  cites  the  limited  autonomy  of  nurses,  teachers,  engineers,  etc.  in 
that  they  generally  operate  in  bureaucratic  institutions  that  limit  the  degree  of  discretion  in  their  work. 

The  term  marginalization  is  a less  strident  term  than  what  some  sociologists  use  to  differentiate 
occupations,  that  is,  full  and  semi-professions,  in  which  semi-professions  (e.g.,  nursing,  teaching)  possess 
several  characteristics  of  the  full  professions  (modeled  by  medicine  and  law),  but  fall  short  in  that  they  are 
employed  by  bureaucracies,  are  often  not  life -long  pursuits,  and  their  knowledge  base  is  short  on  complexity 
(Abbott,  1998). 

Power  or  Market  Models 

The  second  type  of  classifications  is  one  based  on  power  relationships  of  an  occupation  collectively 
with  society,  individual  clients,  government,  and  other  occupations.  Generally,  it  is  the  characteristics  and 
functions  of  the  occupations  within  the  context  of  the  political  economy  that  empowers  occupations  with 
professionalism.  Freidson  (2001),  Larson  (1977),  and  Abbott  (1988)  promote  this  power  model  of 
professionalism  albeit  with  individual  variances. 

Freidson  (1986,  1994,  2001)  develops  his  model  of  the  “third  logic.”  He  proposes  three  different  ways 
to  theorize  about  the  division  of  labor  in  society.  His  “first  logic”  is  -associated  with  the  consumerism  of  the  free 
market  and,  therefore,  it  is  the  consumer  who  controls  the  division  of  labor  through  competitive  market  forces. 
His  “second  logic”  theorizes  how  management  or  bureaucratic  institutions  control  the  division  of  labor  through 
regulation  and  planning.  His  “third  logic"  suggests  that  specialized  skills  and  knowledge  enables  the  profession 
to  more  effectively  control  the  division  of  labor.  Although  none  of  the  “logics”  is  found  in  their  pure  forms  in 
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reality,  they  do  provide  a theory -based  tool  for  analyzing  an  occupation.  His  model  has  several  elements  of 
occupational  control  and  those  occupations  that  can  exercise  these  controls  are  professions. 

Larson  (1977)  emphasizes  the  relationship  of  professions  to  the  market  and  class  systems.  Her 
historical  analysis  focuses  on  how  occupations  in  England  and  the  U.S.  organized  to  capture  a monopoly  in  the 
market  place  in  order  to  secure  an  elevated  position  and  influence  in  society.  She  posits  that  organized  bodies  of 
specialized  practitioners  influence  governmental  bodies  in  order  to  restrict  the  practice  in  a field  of  knowledge 
and  skill  through  legislation. 

Abbott  (1988)  believes  that  the  traditional,  trait -based  models  are  deficient  in  that  they  emphasize 
individual  traits  and  overlook  the  actual  professional  activities  of  these  occupations  within  society.  Though  also 
a power  theorist,  Abbott  considers  his  theory  not  one  of  professionalization  with  its  more  sequential 
perspective.  Abbott’s  system  of  professions  is  more  specific  in  that  it  focuses  on  the  interrelationships  of 
occupational  groups,  their  defining  professional  activities,  and  particularly  the  competition  between  similar 
groups  for  jurisdiction  over  a specialized  body  of  knowledge  and  practice.  The  end  result  of  this  competition 
over  time  results  in  exclusion  of  all  but  those  with  the  jurisdiction  of  the  specialized  skill  and  knowledge. 

What  are  some  of  the  Recognized  Professions? 

Changes  in  society,  technology,  and  bureaucratic  policies  influence  the  status  of  occupations  on  a 
continuum.  Pavalko  (1971,  p.  1 6)  explains  that  occupations  and  professions  are  not  dichotomous  concepts.  It  is 
not  whether  a kind  of  work  is  either  an  occupation  or  a profession,  rather  it  is  the  “degree”  or  “extent”  to  which 
a work  activity  is  a profession.  In  the  final  analysis,  most  sociologists  consider  the  attribution  of  “profession” 
as  an  ideal-type  (Vollmer  & Mills,  1966,  Freidson,  2001)  with  occupations  undergoing  the  dynamic  process  of 
professionalization  (and  in  some  cases,  de-professionalization). 

Recognized  Professions 

Freidson  (1986,  p.  32)  argues  that  “profession”  is  a changing  historical  concept.  As  an  outgrowth  of 
the  medieval  universities,  the  “learned  professions”  were  accorded  special  status  and  included  medicine,  law, 
and  the  clergy  (including  university  professors).  Due  to  the  patronage  of  royalty,  governments,  and  the 
aristocracy,  the  military  was  also  considered  a profession.  However,  with  the  coming  of  capitalist 
industrialization,  the  emerging  middle -class  occupations  began  to  vie  for  the  privilege  and  status  of 
“profession.”  Accountancy,  engineering,  nursing,  school  teaching,  and  social  work  (among  many  others)  were 
subjected  to  analyses  and  case  studies.  In  varying  degrees  and  through  various  processes  of  professionalization 
these  work  activities  became  to  be  considered  “occupational  professions."  Elliot  (1972,  p 14,  32)  differentiated 
the  “status”  professions  of  medicine,  law,  and  clergy  from  the  newer  “occupational  professions”  that  resulted 
from  industrialization. 

Does  IDT  Fit  within  Any  of  the  Professional  Classification  Systems? 

Studying  the  sociological  literature  on  the  professions  can  be,  at  times,  somewhat  confusing.  The 
different  permutations  used  to  analyze  types  of  work,  the  positions  of  occupations  in  society,  and  the 
bureaucracies  that  regulate  and  perhaps  protect  occupations,  leaves  us  searching  for  an  appropriate  model  to 
analyze  IDT  field.  Both  trait-  and  power- model  proponents  look  to  medicine  and  law  as  ideal-typical  “full” 
professions.  By  inference,  then,  those  occupations,  which  do  not  exhibit  all  of  the  traits  of  these  “recognized” 
professions,  fall  somewhat  short  of  the  mark.  However,  more  contemporary  views  (i.e.  Abbott,  1998)  argue  that 
the  characteristics  that  in  the  past  set  law  and  medicine  apart  - “fee  for  service,  internally  enforced  codes,  and 
independent  practice”  - have  changed  with  time  (p.43 1).  It  is  necessary  to  point  out  that  even  with  the  venerable 
“full”  professions  (i.e.  medicine,  law,  etc.)  would  be  considered  as  de -professionalizing  in  certain  aspects  of 
their  fields  based  on  strict  trait  and  power  models  as  guides.  For  instance,  some  doctors  have  unionized  or  now 
work  for  HMOs,  which  eliminates  the  dimension  of  autonomy  for  that  particular  profession.  Therefore,  the 
models  of  medicine  and  law  may  be  inappropriate  analogs  for  IDT. 

Additionally,  we  mentioned  the  skill -based,  economic  model  of  occupations  employed  by  the 
Department  of  Labor.  Using  terms  with  which  our  field  identifies  itself  (ID,  ISD,  IT,  ET)  to  search  the 
Standard  Occupational  Classification  system  database,  which  resulted  in  no  matches  for  our  query.  Therefore, 
the  U.S.  government’s  skill -based  model  is  of  little  use  to  us  in  determining  our  professional  status.  We  are 
uncertain  what  to  make  of  this  finding.  It  may  be  that  the  population  of  IDT  professionals  is  so  comparatively 
small  that  it  fails  to  register  on  a national  scale  analysis. 

If  we  to  look  at  the  common  elements  of  the  trait  models,  we  find  that  IDT  measures  favorably  in  many  of 
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the  characteristics,  which  is  based  on  Pavalko’s  model. 


Theory  and  intellectual  technique: 

Theory  and  research  form  the  basis  of  IDT  practice.  This  is  evidenced 
by  the  robust  content  of  IDT  graduate  programs  of  study. 

Relevance  to  social  issues: 

The  improvement  of  learning,  instruction,  and  performance  represents 
the  focus  of  IDT  in  the  most  socially  relevant  venues,  education  and  industry. 

Training  period: 

IDT  training,  to  this  point,  has  been  the  province  of  the  graduate 
schools  (Gustafson,  2001).  Rigorous  programs  of  studies  at  the  Masters  and 
Doctoral  levels  predominate.  Training  might  also  include  the  concept  of 
credentialism  (e.g.,  licensure  or  certification).  Many  recognized  professions  and 
“emerging”  professions  already  have  licensure  or  certification  processes  to 
recognize  proven  competence  to  work  in  the  field.  IDT  has  yet  to  establish  a 
certification  program.  Movement  has  been  made  to  identify  competencies  of 
IDT  practitioners,  but  proof  of  attaining  competence  through  certification  has 
not  arrived.  Does  IDT  need  certification?  Two  organizations,  ISPI  and  ASTD 
are  developing  certification  programs  for  instructional  design  and  performance 
technology  that  will  be  recognized  by  industry  and  the  government  sector.  Will 
there  be  a certification  for  IDT  developed  suitable  for  the  educational  field? 

Conflicting  opinions  exist,  but  the  dialogue  is  clearly  active. 

Motivation: 

Not  self-serving,  rather  IDT  is  altruistic.  An  examination  of  the  AECT 
code  of  ethics  illustrates  the  principal  concern  of  IDT  is  for  the  client  learner. 

This  is  exemplified  by  Yeaman’s  (2004a,  p.  7)  comment  in  a recent  issue  of 
TechTrends,  “...how  a profession  cares  for  those  who  it  serves  is  what  counts  for 
its  professional  ethics.” 

Autonomy: 

IDT  practitioners  are  normally  employed  by  bureaucracies.  Therefore, 
autonomy  is  limited.  However,  at  the  work  level,  the  level  of  creativity  required 
of  the  IDT  practitioner  is  autonomous  by  nature. 

Sense  of  commitment: 

Many  enter  the  IDT  field  from  other,  perhaps  related,  occupations. 

However,  once  the  one  has  completed  the  extensive  educational  and  training 
program,  a lifelong  commitment  to  the  field  generally  exists. 

Sense  of  community: 

IDT  professionals  can  find  a home  in  at  least  three  international 
organizations,  AECT,  ISPI,  and  ASTD.  These  organizations  serve  as  advocates 
of  the  field  and  foster  research  and  practice.  A professional  community  shares 
developed  knowledge  and  acculturates  its  members  through  periodic 
conferences  and  publication  of  journals  like  ETR&D  and  Tech  Trends. 

Code  of  Ethics: 

IDT  has  a well -developed  code  of  ethics  that  recognizes  that  as 
technology  changes  and  presents  unforeseen  challenges  to  our  ethical  practice, 
that  code  must  be  similarly  dynamic  (Yeaman,  2004b). 

Regarding  the  power  models,  IDT  has  an  interesting  position  with  regard  to  the  division  of  labor,  the  labor 
market  place,  and  “social  closure.”  It  is  a profession  that  is  dependent  upon  other  professions  (teachers  and 
trainers)  to  convey  the  products  and  processes  of  its  industry  (with  perhaps  e-learning  as  an  exception).  IDT  is 
a profession  that  requires  mutual  respect  and  value  of  and  by  other  professions  (if  considered  idiosyncratic  from 
education  in  general  (i.e.,  is  IDT  a “specialization”  within  the  larger  profession  of  education?).  It  can  neither 
dominate  (i.e.,  be  superordinate)  as  in  a hierarchy  nor  subordinate  to  teaching/training  in  its  function  as 
consultant  (internal  to  the  education  process)  (Kerr,  1983).  Perhaps,  it  may  be  that  our  professional 
organizations  are  not  sufficiently  powerful  enough  to  control  access  to  the  IDT  market.  Furthermore,  they  make 
no  attempt  to  regulate  entry  into  the  profession  and  have  only  tangential  influence  on  IDT  programs.  Finally, 
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although  certification  has  long  been  an  issue  with  the  IDT  professionals  within  these  organizations,  it  is  only 
now  that  some  process  for  certification  is  available;  however,  it  is  only  available  for  a specialty  of  performance 
technology  (Davidson-Shivers  & Rasmussen,  in  press). 

Are  There  Specializations  within  the  IDT  Field? 

We  are  uncertain  as  to  why  either  the  trait  or  power  model  theorists  did  not  include  specialization  as 
one  of  their  criteria  for  professional  status.  As  Parelius  and  Parelius  (1987,  p.203)  explain  “ . . . there  can  be 
little  doubt  that  specialization  is  conducive  to  the  development  of  expertise  and  that  expertise  is  central  to  the 
professional  standing  of  an  occupation.”  Although  the  classification  systems  vary,  a common  factor  among 
professions  is  that  of  specialization.  A survey  of  the  most  traditionally  recognized  professioas  (i.e.,  medicine, 
law,  accountancy,  and  engineering)  reveals  that  the  differentiation  of  skills  occurs  commensurate  with  the 
increase  in  that  field’s  body  of  knowledge  and  historically  as  the  occupations  professional  status  strengthened 
(Abbott,  1988).  The  field  of  IDT  is  no  less  complex;  this  complexity  is  based  on  its  broad  application  as  well  as 
the  technological  advances  that  are  associated  with  IDT  work. 

Richey,  Fields,  and  Foxon  (2001)  discuss  the  nature  of  IDT  specialization  and  suggest  three  general 
areas:  analysis  and  evaluation,  e-learning,  and  project  management.  Other  areas,  including  those  of  designing 
and  developing,  might  also  be  included.  For  instance,  Davidson  (1987)  suggested  that  designers  may  focus  not 
only  on  different  aspects  of  the  design  process  (e.  g,  analysis,  design,  development,  implementation — training 
and  instruction,  and  evaluation),  but  could  also  specialize  by  a particular  technology  or  delivery  (e.g., 
videography,  platform  training,  computers,  etc.),  or  be  oriented  toward  a particular  setting  (e.g.,  business  and 
industry,  military,  health  care,  education,  etc.). 

Even  though  specialization  has  been  discussed  in  the  IDT  literature  over  the  years,  once  again  there  is 
no  process  in  place  for  recognizing  or  ordering  specializations  within  our  field  and  no  organization  to  monitor 
this  process.  By  contrast,  with  recognized  traditional  professions,  an  overarching  organization  typically  controls 
and  monitors  the  maturation  of  the  subspecialties. 


Summary 

There  are  several  equally  supported  perspectives  on  what  identifies  professions.  Even  though  there  is 
disagreement  as  to  what  determines  the  professional  status  of  a field,  it  is  important  to  not  disregard  these 
perspectives,  especially  in  considering  the  IDT  field  as  a profession.  By  most  trait  or  power/market  models,  the 
IDT  field  as  a whole  is  a profession. 

The  degree  to  which  an  occupation  is  considered  a profession  is  very  subjective.  Consequently,  the 
appropriateness  of  ascribing  a descriptor  or  adjective  to  the  term  ‘‘profession”  to  assign  status  seems,  at  this 
point  in  the  discourse,  less  productive  than  an  examination  of  the  strengths  and  weaknesses  of  the  field  in  each 
of  the  characteristics  of  a profession  and  its  relationship  with  other  professions  and  its  clients.  Of  more 
importance,  IDT  professionals  should  concentrate  on  refining  the  definition  of  the  field,  seeking  consensus  as  to 
its  identity  and  the  names  by  which  it  is  called,  and  continue  strengthening  the  knowledge  base.  However,  these 
tasks  cannot  be  accomplished  by  the  individual  practitioners;  instead  they  must  be  accomplished  by  the 
collective  efforts  of  organizations,  which  represent  the  field  (i.e.,  AECT,  IBSTPI,  ISPI,  etc.).  Hopefully,  this 
paper  will  help  trigger  such  efforts  to  begin,  once  again. 
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Abstract 

The  presentation  describes  the  revisions  to  a course  and  the  resulting  student  attitudes  and  learning. 
Learning  Tools  was  revised  in  2003 from  oncampus  only  to  both  oncampus  and  online  delivery.  Revisions  were 
made  by  standardizing  the  two  versions,  updating  the  technology  applications  presented,  and  modifying  the 
instructional  strategies  used.  These  changes  were  based,  in  part,  on  the  evolving  technology  and  survey  data 
(former  students  and  instructors).  The  results  of  student  opinion  surx’ey  will  be  presented  as  well  as  suggestions 
for  future  directions  of  the  course. 


Introduction 

Christenson  (2002)  suggests  that  teaching  technology  has  a positive  effect  on  teacher  opinions, 
including  reduced  computer  anxiety  and  increased  computer  enjoyment,  but  noted  a time  lag  for  positive  effects 
on  students  with  technology  tools.  Factors  that  may  affect  teaching  technology  and  corresponding  student 
opinions  include  learner  and  instructor  anxiety,  experience  level,  and  the  evaluation  instrument  itself.  For 
instance,  student  anxiety  toward  technology  is  increased  for  those  with  little  or  no  computer  experience 
(Necessary  & Christensen,  1996)  or  who  are  required  to  take  a course  rather  than  choosing  it  as  an  elective 
(Parish  & Necessary,  1996).  Khine  (2001)  found  that  males  are  more  confident  and  feel  less  anxiety  than 
females  when  using  computers,  perhaps,  because  they  have  more  computer  experience.  Marginal  differences 
showed  that  females  like  to  use  computers  and  appreciate  their  usefulness,  but  still  exhibit  anxiety  toward  using 
computers. 

Abbott  and  Faris  (2000)  suggest  that  positive  attitudes  toward  instruction  may  be  because  students 
teach  themselves;  instructors  foster  collegial  atmospheres  in  which  students  complete  course  requirements  and 
are  committed  to  providing  successful  technology  experiences.  Sweaney  (2001)  suggests  that  students  are  more 
likely  to  learn  technology  by  being  able  to  play  with  the  technology.  Furthermore,  Koltich  (1999)  states  that 
mutual  respect  by  teacher  and  student  facilitates  positive  attitudes  and  learning.  A somewhat  hidden  implication 
for  studying  attitudes  toward  learning  technology  may  lie  in  the  instrument  itself.  Research  findings  by  Kolitch 
(1999)  using  the  Student  Evaluation  of  Instruction  suggest  that  it  may  lead  students  toward  opinions  that  reject 
alternative  instructional  methods  involved  with  learning  technology. 

Historical  Perspective  to  Changes  to  the  Learning  Tools  Course 

Learning  Tools  is  a graduate  course  designed  to  introduce  students  to  several  basic  media  tools.  Its 
purpose  is  to  assist  students  with  their  coursework  and  ultimately  in  the  work  place  (2002-2003  Undergraduate 
& Graduate  Bulletin).  Learning  Tools  has  been  taught  as  a weekend  course  for  1 credit  and  scored  on  a pass/fail 
basis  for  over  the  last  12  years.  However,  Learning  Tools  has  evolved  over  a decade  with  the  changes  in  the 
types  of  technology  taught  and  how  it  has  been  implemented  (Davidson-Shivers,  Jackson,  & Wimberg,  2003). 
Changes  in  the  instructional  strategies  and  delivery  systems  were  based  on  the  practical  matter  of  needing  an 
online  version  of  the  course,  the  evolving  technology  related  to  the  IDT  field,  survey  data  of  former  students 
and  instructors  (Wimberg  & Jackson,  2003),  learning  psychology  principles  that  advocate  practice  and  active 
participation  (Driscoll,  2001;  Mayer,  2003;  Ormrod,  2003),  and  guidelines  for  teaching  technology  tools 
(Davidson-Shivers,  Jackson,  & Wimberg).  A table  of  the  chronology  was  shown  at  the  presentation. 

Because  the  changes  were  significant,  it  was  decided  to  document  the  changes  made  and  address  what 
effect,  if  any,  they  had  on  student  learning  and  attitudes  toward  the  course.  For  example,  in  the  past,  it  has  been 
taught  by  an  instructor  in -charge  with  the  use  of  additional  facilitators  and  was  delivered  as  a weekend  course. 
The  delivered  only  as  an  oncampus  course  with  multiple  instructors  teaching  the  contents  during  two -hour 
intervals  over  a term.  Fall  2003  marked  the  first  time  it  was  taught  online  using  web  resources,  with  one 
instructor  being  its  facilitator  while  a second  offering  provided  an  on-campus  course  facilitated  by  two  graduate 
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students.  While  the  University  requires  that  online  courses  be  equivalent  in  content  and  course  requirements  to 
their  on-campus  counterparts,  some  differences  occurred  in  order  to  take  advantages  of  the  Web  environment 
and  instill  some  sense  of  community  among  learners. 

Methodology 

Participants 

The  participants  in  this  study  were  graduate  students  (A=20)  in  Master’s  and  Doctoral  IDT  program  at 
a southeastern  university.  These  students  have  varied  educational  and  professional  backgrounds,  often  fully 
employed.  Approximately  equal  numbers  of  males  and  females  (h=10)  were  enrolled  in  the  online  version 
whereas  the  oncampus  version  («=10)  had  only  two  males. 

The  changes  in  the  choices  of  software  were  based,  in  part,  on  data  gathered  from  the  student  and  instructor 
survey  (Wimberg  & Jackson,  2003).  The  most  significant  perception  gathered  from  this  survey  was  in  the  area 
of  requiring  assignments  for  a grade.  Both  instructors  and  students  agreed  that  adding  an  assignment  for  a grade 
would  improve  the  delivery  of  the  learning  tools  couise. 

Course  Redesign  & Revision  Guidelines 

The  underlying  assumption  for  these  changes  is  that  students  are  becoming  sophisticated  in  using 
technology;  tools  taught  10  years  ago  are  now  considered  prerequisites  for  the  current  course  offerings.  A 
second  reason  for  change  is  to  keep  students  current  on  new  technology  tools  as  they  emerge.  The  redesign  and 
revision  of  the  Learning  Tools  course  resulted  in  the  oncampus  version  becoming  a web-enhanced  instruction 
(WEI)  and  the  addition  of  an  online  version  with  web-based  instruction  (WB1).  The  following  guidelines  were 
used  in  the  development  and  implementation  for  both  the  WB1  and  WEI  versions: 

1)  Provide  an  overview  of  learning  tools  for  students  to  acquire  basic  skills  rather  than 

proficiency; 

2)  Provide  meaningful  assignments  as  indicators  of  knowledge  gained; 

3)  Provide  opportunities  for  collaboration  and  questions  about  assignments.  A threaded 

discussion  called  “The  Student  Lounge"  was  provided  for  both  versions. 

4)  Class  size  was  not  to  exceed  12  students  for  adequate  management  and  fostering  of  a 

collegial  atmosphere. 

5)  Explain  that  students'  roles  were  as  self -regulated  learners  at  the  beginning  of  both 

versions. 

6)  Encourage  students  to  search  for  other  tutorials  and  materials  through  the  Web,  library,  etc. 

7)  Explain  that  instructor  is  a facilitator,  not  sole  knowledge  authority  or  provider. 

8)  Instructors  (faculty  member  and  graduate  assistants)  assisted  students  when  needed  through 

emails,  office  hour  meetings,  and  phone  conversations;  and 

9)  Both  versions  were  developed  and  delivered  by  one  faculty  member  and  two  graduate 

assistants  who  actively  provided  student  support. 

Learning  Tools  Content  & Instructional  Materials 

The  current  versions  contain  eight  technology  sessions  as  follows:  a)  An  introduction  to  Windows  basics,  the 
Web,  and  online  library  resources,  b)  MS  Excel  & Access,  c)  MS  PowerPoint,  d)  Adobe  Acrobat  Reader,  Inspiration, 
and  media  players,  e)  Adobe  Photoshop  f)  Windows  Sound  Recorder  and  Cool  Edit,  g)  Dreamweaver  & Websites,  and 
h)  Course  Wrap-up  and  Evaluations.  Students  were  asked  to  purchase  a textbook  related  to  MS  Office  XP  or  2000, 
depending  on  the  software  installed  on  their  computers.  In  addition,  a list  of  online  reference  materials  and  tutorials  were 
provided  for  both  versions  of  the  course.  Short  biographies  of  students  and  instructor  or  teaching  assistants  were 
provided  for  both  versions. 

Text -based  and  PowerPoint  lectures  were  supplied  to  the  WBI  version,  and  "live  lectures"  accompanied  by  the 
same  PowerPoint  materials  were  presented  to  the  WEI.  Both  courses  allowed  for  students  and  instructor/teaching 
assistants  to  communicate  with  each  other  through  e-mail  and  threaded  discussion.  In  addition,  students  had  their  own 
threaded  discussion  (entitled  Student  Lounge)  in  which  students  could  post  helpful  suggestions  or  ask  questions  to  each 
other.  The  teaching  assistant  monitored  them  as  well  in  order  to  alleviate  any  frustrations  due  to  technical  difficulties. 

Instruments 

Three  self-report  instruments  were  used:  Former  Student  and  Instructor  Survey,  Proficiency  Checklist, 
and  Student  Attitudes  toward  Learning  Tools  Course.  In  addition,  extant  data  from  course  records  provided  the 
information  for  the  changes  recorded  over  the  last  decade.  Extant  Data  , shown  in  the  presentation,  were 
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discovered  by  examining  old  course  records  and  interviews  with  faculty  members  who  had  taught  the  course  in 
the  past. 

The  Former  Student  and  Instructor  Surveys  were  developed  by  one  of  the  researchers  and  contained 
thirteen  quantitative  items.  Table  1 shows  the  responses  by  former  students  and  Table  2 shows  response  of 
former  instructors.  This  survey  also  included  eleven  open-ended  questions  included  a description  of  the  course 
from  the  student’s  or  instructor’s  perspective,  how  well  he  or  she  thought  the  class  prepared  the  student,  and  a 
discussion  of  the  chosen  topics. 


Table  1 .Results  of  Former  Student  Perceptions  of  the  Learning  Tools  course 


Question 

Continuum 

Average 

1.  How  much  did  you 
learn  about  learning  tools 
(software)  in  ISD  600? 

1 2 3 4 5 

nothing  not  much  a moderate  a good  a lot 

amount  amount 

2.6 

2.  How  well  did  ISD  600 
prepare  you  for  IDD 
classes? 

1 2 3 4 5 

not  well  fairly  well  adequately  very  well  extremely  well 

at  all 

2.2 

3.  How  well  did  ISD  600 
prepare  you  for  your 
current  job? 

1 2 3 4 5 

not  well  fairly  well  adequately  very  well  extremely  well 

at  all 

1.9 

4.  How  beneficial  was 
ISD  600? 

1 2 3 4 5 

not  at  all  fairly  beneficial  very  beneficial  extremely 

beneficial  beneficial 

2.6 

5.  How  effective  was  the 
instruction  in  ISD  600? 

1 2 3 4 5 

not  fairly  effective  very  effective  extremely  effective 

effective 

2.3 

6.  How  effective  was  the 
design  of  the  course 
regarding  the  methods 
used  to  teach  the  course? 

1 2 3 4 5 

not  fairly  effective  very  effective  extremely  effective 

effective 

2.2 

7.  If  ISD  600  were  an 
elective,  I would 
recommend  other 
students  take  the  class. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

2.7 

8.  The  class  required  I 
put  forth  much  effort. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

2.4 

9.  The  topics  taught  were 
too  difficult. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

4.2 

10.  The  topics  taught 
were  too  easy. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

3.2 

1 1 . The  class  would  have 
been  better  if  I had  to 
produce  assignments  for 
a grade. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

2.5 

12.  The  amount  of 
exposure  to  various 
learning  tools  was 
sufficient. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

2.7 

13.  ISD  600  was  a waste 
of  time? 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

2.9 
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Table  2.  Results  of  Former  Instructor  Perceptions  of  the  Learning  Tools  course  survey 


Question 

Continuum 

Average 

1.  How  much  do  you 
think  students  learned 
about  learning  tools 
(software)  in  ISD  600? 

1 2 3 4 5 

nothing  not  much  a moderate  a good  a lot 

amount  amount 

3.2 

2.  How  well  do  you 
think  ISD  600  prepared 
students  for  IDD 
classes? 

1 2 3 4 5 

not  well  fairly  well  adequately  very  well  extremely  well 

at  all 

2.7 

3.  How  well  do  you 
think  ISD  600  prepared 
students  for  their  current 
jobs? 

1 2 3 4 5 

not  well  fairly  well  adequately  very  well  extremely  well 

at  all 

2.2 

4.  How  beneficial  do  you 
think  ISD  600  was  for 
students? 

1 2 3 4 5 

not  at  all  fairly  beneficial  very  beneficial  extremely 
beneficial  beneficial 

3.0 

5.  How  effective  was  the 
instruction  in  ISD  600? 

1 2 3 4 5 

not  fairly  effective  very  effective  extremely 

effective  effective 

2.8 

6.  How  effective  was  the 
design  of  the  course 
regarding  the  methods 
used  to  teach  the  course? 

1 2 3 4 5 

not  fairly  effective  very  effective  extremely 

effective  effective 

2.7 

7.  If  ISD  600  were  an 
elective,  I would 
recommend  other 
students  take  the  class. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

3.5 

8.  The  class  required  I 
put  forth  much  effort. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

2.0 

9.  The  topics  taught  were 
too  difficult. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

2.2 

10.  The  topics  taught 
were  too  easy. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

2.2 

1 1 . The  class  would  have 
been  better  if  students 
had  to  produce 
assignments  for  a grade. 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 

agree 

2.7 

12.  The  amount  of 
exposure  to  various 
learning  tools  was 
sufficient. 

1 2 3 4 5 

strongly  disagree  somewhat  agree  agree  strongly  agree 
disagree 

2.8 

13.  ISD  600  was  a waste 
of  time? 

1 2 3 4 5 

strongly  agree  somewhat  agree  disagree  strongly  disagree 
agree 

2.3 

The  Proficiency  Checklist  was  developed  by  one  of  the  researchers  and  asked  students  at  the  beginning 
of  the  course  to  report  their  proficiency  related  to  various  software  applications;  it  contained  twenty  items. 
Students  were  asked  to  check  the  response  that  most  closely  resembled  their  computer  and  Web  capabilities. 
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Students  were  given  the  following  choices:  1)  F=  Familiar  = I have  only  heard  of  this  software,  2)1  = 
Intermediate  = I have  used  this  software  often,  but  I am  still  unsure  of  some  functions,  3)  P = Proficient  = I 
know  this  software  well  enough  to  teach  it  in  its  entirety.  The  following  software  was  included  in  the  checklist: 
Windows  Basics,  Internet  & WWW,  University  Online  Library,  several  Microsoft  applications.  Media  Players, 
Photo,  Sound,  and  Website  Editors,  and  Web  Management  Systems.  See  Table  3 for  the  results. 

Table  3 . Results  of  Student  Proficiency  Checklist  (WEI  version  only) 


Software 

Familiar 

Intermediate 

Proficient 

Windows  Basics 

1 

8 

1 

MS  Word 

7 

3 

Internet  & WWW 

9 

1 

USA  Library  Online 

1 

9 

MS  Excel  spreadsheets 

2 

6 

2 

MS  Access  databases 

5 

5 

MS  PowerPoint  slideshows 

1 

6 

3 

Adobe  Acrobat  Reader  .pdf 

2 

7 

1 

Inspiration 

5 

4 

1 

Windows  Media  Players 

3 

7 

Other  Media  Players 

5 

5 

Adobe  Photoshop  images 

8 

2 

Other  Photo  editing 

8 

2 

Windows  Sound  Recorder 

8 

2 

Cool  Edit 

9 

1 

Other  Audio  File  Editor 

10 

Dreamweaver 

9 

1 

Other  Webpage  Composer 

9 

1 

eCompanion 

6 

4 

eColleae 

4 

6 

N=  10 


The  final  instrument  was  the  Student  Attitudes  toward  Learning  Tools  Course,  which  contained  thirty- 
seven  items  and  asked  students  to  report  their  opinions  about  the  course.  Students  reported  experience  levels 
with  computers  and  online  courses,  preferences  of  topics  and  instructional  pace,  expectations  of  instructor,  and 
difficulties  with  assignments. 

Procedures 

The  courses  were  conducted  in  fall  2003.  The  online,  or  WBI,  version  ran  the  entire  semester  whereas 
the  on  campus,  or  WEI,  version  was  conducted  during  the  last  eight  weeks  of  the  semester  (due  to  conflicts  with 
another  weekend  course  held  on  Friday  afternoons).  To  accommodate  for  the  compressed  time  of  the  WEI 
version,  students  were  allowed  to  complete  the  course  in  two  terms  rather  than  in  one  semester. 

The  Proficiency  Checklist  was  administered  within  the  first  two  weeks  of  either  version.  Data  from  this 
checklist  were  reviewed  at  the  time  of  its  administration  in  order  to  make  any  necessary  adjustments  to  the 
course.  However,  the  only  changes  to  types  of  software  were  due  to  the  availability  to  free  trial  versions  of  the 
software  rather  than  due  to  proficiencies.  A second  change  to  the  WBI  version  was  to  open  the  last  four  sessions 
for  the  entire  session  enabling  students  to  access  the  last  four  sessions  at  any  time  during  the  term  rather  than 
only  having  access  to  a session  during  the  time  scheduled  within  the  web  course. 

The  students  were  required  to  attend  the  sessions  and  were  given  time  to  complete  assignments  (two 
weeks  for  the  WBI  and  one  week  for  the  WEI  version);  assignments  were  submitted  either  through  an  online 
drop  box  to  the  instructor  or  via  email.  Some  assignments  for  the  WBI  were  also  to  respond  to  either  document 
sharing,  threaded  discussions,  and/or  locating  websites.  They  received  feedback  from  the  instructor  on  their 
assignments  one  week  after  completion  based  on  a range  of  scores  from  satisfactory  plus  (S+)  to  unsatisfactory 
minus  (U-). 

The  Student  Attitudes  toward  Learning  Tools  Course  was  given  at  the  end  of  the  course  during  the  wrap-up 
session  in  both  WEI  and  WBI.  All  data  collected  were  analyzed  after  the  course  had  ended  and  final  grades 
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were  posted. 


Discussion  and  Summary  of  the  Results 

We  are  happy  to  report  that  all  students  passed  the  course  (both  versions)  with  a satisfactory  grade. 
Their  scores  were  due  in  part  to  meeting  the  requirements  of  the  course,  but  also  to  their  level  of  proficiency  in 
using  computers  and  the  Web.  Most  students  had  had  some  experience  with  online  courses  as  required  to  some 
degree  by  other  courses  in  their  IDT  program.  (Refer  back  to  Table  3 for  the  student  proficiency  checklist 
results.)  However,  when  dealing  with  audio,  photo,  sound,  and  website  editors,  most  of  the  students  reported 
only  a familiarity  with  such  software.  In  addition,  the  WEI  students  reported  low  proficiencies  in  the  web 
management  system. 

The  results  from  the  Student  Attitude  toward  a Learning  Tools  Course  administered  at  the  end  of  the 
semester  indicated  that  students  reported  being  proficient  in  using  computers  (70%  in  WEI  and  89%  in  WBI) 
and  students  in  both  versions  had  high  levels  of  comfort  with  using  the  Web  (100%)  and  60%  in  WEI  and  70% 
in  WBI  had  previously  taken  an  online  course.  The  results  of  the  Student  Attitude  toward  Learning  Tools 
Course  questionnaire  suggested  that  only  80%  of  the  WEI  students  felt  comfortable  with  that  delivery  format 
whereas  100%  in  the  WBI  student  felt  comfortable.  100%  in  WEI  reported  agreement  that  course  topics  were 
relevant  whereas  89%  in  the  WBI  agreed,  with  11%  strongly  disagreeing.  Those  disagreeing  were  most  likely  to 
have  been  very  experienced  with  computers  and  thought  that  the  expectations,  content,  and  pace  of  the 
beginning  session  were  set  too  low  for  them.  It  should  be  noted  that  because  we  were  unable  to  gain  information 
about  the  online  students  and  their  proficiency  in  the  WBI  until  it  began  and  to  be  on  the  conservative  side,  we 
used  the  first  session  to  cover  some  of  the  basics  with  MS  Office  and  the  Web.  Perhaps  the  'bar'  was  set  too  low 
for  this  session  and  for  the  types  of  students  who  took  the  online  version  of  this  course. 

We  also  found  that  80%  in  both  the  WEI  and  the  WBI  liked  the  student  lounge  (their  own  threaded 
discussion)  option  and  56%  in  WBI  also  liked  the  option  to  discuss  assignments.  The  WEI  students  (50%) 
reported  that  the  instructor  should  be  expected  to  help  with  technical  problems  whereas  70%  in  the  WBI 
disagreed  with  that  statement.  Again,  this  finding  may  reflect  a difference  in  experience  level  of  those  students 
who  opt  for  the  oncampus  version  vs.  the  online  versions.  Students  in  both  versions  reported  finding  the  online 
tutorials  were  helpful  and  students  of  both  versions  agreed  that  the  lectures  were  helpful.  Almost  all  WEI 
students  suggested  that  the  units  of  instruction  were  clear.  Most  in  both  versions  reported  overall  satisfaction 
with  the  course.  The  course,  both  versions,  appears  to  be  successful  based  on  student  opinion.  Table  4 shows 
the  results  of  the  entire  survey. 

Table  4.  Results  of  Student  Attitudes  toward  a Learning  Tools  Course 


Question 

WEI  Mean 

WBI  Mean 

1 

I have  a great  deal  of  computer  experience. 

2.80 

3.22 

2 

I have  a great  deal  of  prior  knowledge  or  experience  in  instructional 
design. 

2.00 

2.33 

3 

I am  very  comfortable  working  on  the  Internet  and  the  WWW. 

3.30 

3.67 

4 

Technical  problems  frustrate  me. 

2.90 

2.44 

5 

The  pace  of  the  class  was  too  rapid. 

1.87 

1.33 

6 

I think  that  I worked  hardere  in  this  class  than  if  I were  in  the  online 
version  (classroom  version). 

2.93 

1.67 

7 

I feel  that  as  a student,  I had  enough  control  over  my  learning  in  this  class. 

3.20 

3.44 

8 

The  assignments  for  the  units  were  not  that  difficult. 

2.80 

3.11 

9 

The  feedback  I received  on  assignments  was  sufficient  for  this  class. 

3.25 

3.22 

10 

Removed. 

11 

I liked  the  student  lounge  option. 

3.11 

2.86 

12 

I have  taken  a web-enhanced  (web-based)  class  prior  to  this  one. 

2.60 

3.11 

13 

I expected  immediate  help  from  my  instructor  when  I had  problems. 

2.60 

2.22 

14 

I contacted  my  instructor  during  the  semester  about  a problem  or  question. 

6 Yes  4No 

7Yes  2No 

14a 

The  instructor  or  the  teaching  assistants  helped  me  with  my  problem  or 
question. 

3.83 

3.63 
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fgm 

The  instructor  or  the  teaching  assistants  provided  me  with  immediate  help. 

3.67 

3.38 

m 

I am  comfortable  with  web-enhanced  (web-based)  instruction. 

2.89 

3.63 

m 

The  topics  in  this  course  are  relevant. 

3.60 

3.33 

m 

The  topics  in  this  course  will  be  helpful  to  me. 

3.70 

3.22 

m 

I liked  being  able  to  work  at  my  own  pace  on  course  assignments. 

3.80 

3.89 

H 

I liked  that  each  unit  has  a precise  start  and  stop  date. 

3.70 

2.56 

20 

I liked  being  able  to  access  all  of  the  units  at  once  without  waiting  for  the 
unit  start  date. 

3.00 

3.78 

■ 

I contacted  eCollege  Help/support  for  technical  problems 

OYes  lONo 

3Yes  6No 

21a 

I was  able  to  get  help  with  technical  problems  from  the  eCollege/Help 
support  desk. 

N/A 

4.0 

22 

It's  realistic  to  expect  my  instructor  to  respond  to  my  email  concerns 
within  a couple  of  hours. 

1.75 

1.75 

I expected  my  instructor  to  help  me  with  technical  problems. 

2.67 

1.75 

I found  the  online  tutorials  helpful  (webliography). 

3.33 

3.00 

I found  the  lectures  (text  or  PowerPoint)  helpful. 

3.20 

3.22 

The  assignments  clearly  identified  the  tasks  to  do  for  each  unit. 

9 Yes  INo 

8 Yes  INo 

I had  very  few  technical  problems. 

3.10 

3.33 

I had  very  few  problems  with  assignments. 

3.20 

3.44 

The  assignments  took  a long  time  to  complete. 

1.90 

1.56 

I needed  more  interaction  with  the  instructor  and  the  other  students. 

2.10 

1.67 

m 

Overall,  I am  satisfied  with  this  course. 

3.20 

3.56 

■ 

l=Strongly  Disagree  2=Disagree  3=  Agree  4=  Strongly  Agree 

N=10 

N=  9 

Future  Implications  Based  on  the  Results 

One  of  the  implications  for  this  course  is  that  the  types  of  software  and  delivery  of  the  course  will 
continue  to  evolve.  We  anticipate  that  future  versions  of  the  course  will  involve  teaching  about  creating  and 
using  video  for  course  projects  as  well  as  new  Web  features,  such  as  blogging,  as  they  become  available.  The 
course  may  also  involve  the  use  of  other  types  of  equipment  such  as  PDAs,  cell  phones,  etc.  as  well  as  video 
and  audio  conferencing.  Again,  introduction  to  software  and  hardware  advances  will  be  based  on  the 
availability  of  new  versions,  new  innovations,  and  how  student  proficiency  advance.  Although  it  is  too  early  to 
predict  or  prescribe,  over  time  we  may  find  out  that  those  who  take  Learning  Tools  online  may  have  higher 
proficiency  levels  than  those  who  prefer  the  oncampus  version  and,  hence,  prefer  topics  that  require  more 
advanced  computing  skills  for  each  session.  However,  much  will  depend  on  the  level  of  lowest  common 
denominator,  effect  of  the  computer  equipment,  high-speed  accessibility  for  the  participants. 

The  guidelines  on  which  the  on-campus  and  online  versions  were  based  seemed  to  be  appropriate.  The 
two  versions  were  kept  in  alignment  in  terms  of  what  was  taught  and  required  of  student  participants.  In 
addition,  the  use  of  the  Web  resources  and  materials  in  both  allowed  for  ease  in  accessing  and  using  tutorials 
and  free  trial  versions.  The  class  size,  being  kept  to  a small  number,  was  manageable  for  introducing 
technology;  having  teaching  assistants  also  helped  in  terms  of  providing  timely  feedback  and  troubleshooting. 
We  would  advocate  that  such  courses,  especially  when  taught  in  a virtual  classroom,  be  kept  to  a minimum 
number  of  students.  Although  the  interaction  required  among  students  for  both  versions  was  kept  to  a minimum, 
they  seemed  to  develop  a sense  of  community.  Because  the  WEI  students  had  face-to-face  interaction,  the 
student  lounge  threaded  discussion  may  have  been  less  important  or  necessary  for  them.  However,  students  in 
both  versions  seemed  to  enjoy  this  threaded  discussion  and  those  in  the  WBI  enjoyed  discussing  the 
assignments  amongst  themselves.  Hence,  we  think  that  the  amount  of  interaction  was  set  at  the  appropriate 
level. 


263 


References 

Abbott,  J.  A.,  & Faris,  S.  E.  (2000).  Integrating  technology  into  preservice  literacy  instruction:  a survey  of 
elementary  education  students’  attitudes  toward  computers.  Journal  of  Research  on  Computing  in 
Education , 33(2),  149-162. 

Christensen,  R.  (2002).  Effects  of  technology  integration  education  on  the  attitudes  of  teachers  and  students. 

Journal  of  Research  on  Technology  in  Education,  34(4),  411-434. 

Davidson-Shivers,  G.  V.,  Jackson,  M.  K.,  & Wimberg,  J.  E.  (2003,  November).  Design  & Delivery  of  an 
Evolving  Technology  Course:  A Decade  of  Changes.  A paper  presentation  at  the  Mid-South 
Educational  Research  Association,  Biloxi,  MS. 

Davidson-Shivers,  G.  V.,  Wimberg,  J.  E.,  & Jackson,  M.  K.  (2004,  April).  Changes  to  a Learning  Tool  Course. 

Paper  presented  at  the  USA  Research  Forum,  Mobile,  AL. 

Rhine,  M.  S.  (2001).  Attitudes  toward  computers  among  teacher  education  students  in  Brunei  Darussalam. 

International  Journal  of  Instructional  Media,  28(3),  147-153. 

Kolitch,  E.,  & Dean,  A.V.  (1999).  Student  ratings  of  instruction  in  the  USA:  Hidden  assumptions  and  missing 
conceptions  about  ‘good’  teaching  practices.  Studies  in  Higher  Education,  24(1),  27-42. 

Necessary,  J.  R.,  & Parish,  T.  S.  (1996).  The  relationships  between  computer  usage  and  computer-related 
attitudes  and  behaviors.  Education,  116(3),  384-387. 

Parish,  T.  S.,  & Necessary,  J.  R.  (1996).  An  examination  of  cognitive  dissonance  and  computer  attitudes. 
Education,  116(4),  565-567. 

Sweaney,  A.  L.  (2001).  Computer  experience  and  skills  of  family  and  consumer  sciences  undergraduates  and 
professionals.  Education,  121(4),  773-781. 

Wimberg,  J.  E.,  & Jackson,  M.  K.  (2003).  A qualitative  analysis  of  the  perceptions  of  instructors  and 

students  for  improving  the  delivery  of  instruction  for  learning  tools . Course  project  and  paper  for  IDE 
631  Qualitative  Research.  Mobile,  AL:  University  of  South  Alabama. 


264 


Investigation  of  Source  of  Motivation  in  a Hybrid  Course 

Omer  Delialioglu 
Middle  East  Technical  University 

Abstract 

The  current  study  investigated  the  sources  of  motivation  for  learning  in  a hybrid  course.  The  subjects 
of  the  study  were  25  students  taking  a hybrid  course  that  was  designed  and  developed  covering  computer 
networks  topics.  An  interview  form  that  revealed  answers  about  the  motivation  source  of  the  students  was 
developed  and  used  in  the  study.  One  on  one  interviews  were  made  with  the  students.  Students’  answers  to  the 
questions  were  recorded  and  transcribed.  The  interview  data  for  the  students  were  analyzed  by  content 
analysis.  Students’  responses  were  interpreted  and  categorized  into  two  types  of  motivation,  extrinsic  and 
intrinsic.  Results  indicated  that  intrinsic  motivation  and  internally  rewarded  learning  is  the  key  element  of  web 
based  instruction  and  hybrid  courses.  Interviews  revealed  that  students  with  extrinsic  motivation  are  more 
prone  to  loosing  motivation.  It  was  seen  that  some  students  in  the  hybrid  course  with  extrinsic  motivation  lost 
their  motivation  and  will  to  learn  easily  by  external  factors,  and  were  frustrated  by  the  course  content.  On  the 
other  hand,  students  with  internal  motivation  were  more  aware  of  objectives  of  the  course  and  had  the  ability  to 
plan  and  evaluate  their  own  learning.  An  in  depth  analysis  of  students'  sources  of  motivation  in  a hybrid  course 
on  computer  networks  revealed  that  intrinsic  motivation  plays  a more  important  role  than  extrinsic  motivation 
does. 


Introduction 


WBI  and  Hybrid  Instruction 

Web  Based  Instruction  (WBI)  was  defined  as  a learning  environment  in  which  learning  was  fostered 
and  supported  through  the  use  of  the  attributes  and  resources  of  the  World  Wide  Web  (Khan,  1997).  The  major 
advantage  of  WBI  was  stated  as  being  able  to  communicate  with  any  person  and/or  access  many  resources 
independent  from  time  and  distance  (Hill,  1997).  This  structure  was  suitable  for  constructivism  because  of  the 
time  independency  and  freedom  to  access  learning  material  at  will.  To  understand  the  effectiveness  of  the  WBI 
environment  several  instructional  models  were  developed.  The  models  of  Reeves  and  Reeves  (1997),  Caladine 
(1999)  and  Welsh  (1997)  were  important  guidelines  for  the  WBI  designers. 

The  idea  behind  a hybrid/blended  instruction  is  to  redesign  the  instruction  to  use  the  advantages  of 
both  face  to  face  and  online  modes  of  instruction.  Some  of  the  activities  which  students  previously  did  in 
classroom  or  laboratory,  such  as  listening  to  lecture,  taking  notes,  quizzes,  pre-lab  assignments  could  be  done 
online.  This  change  could  have  positive  effects  on  teaching  resources  like  teachers  workloads,  accommodating 
various  learning  styles  and  hours  of  classroom  time,  and  course  budget.  Actually  hybridizing  different  methods 
of  course  delivery  was  not  a new  idea.  Clark  (2002)  commented  that  hybridizing  has  deep  roots  that  lay  back  to 
times  where  books,  videos  and  print  materials  were  used  as  an  integral  part  of  the  instruction.  Hybridizing  could 
be  understood  as  “mixing’'  or  “blending”.  In  general  terms  we  can  refer  to  hybrid  instruction  as  the  blending  of 
classroom-based  instruction  with  instruction  via  other  media. 

There  are  few  studies  on  hybrid  instruction.  Most  of  them  point  towards  the  advantages  of  these 
courses.  These  advantages  mainly  come  from  the  online  enhancement  of  the  face  to  face  courses.  These 
advantages  were  listed  by  Valerie  Landau  (2002): 

■ Accessible  handouts,  syllabi  and  notes  online,  cutting  down  time  and  resources  in 
photocopying. 

■ Allows  peer  to  peer  collaboration  on  projects,  helping  to  facilitate  and  document  group 
work. 

■ Allows  automatic  grading  of  quizzes  and  tests. 

■ Allows  students  to  discuss  topics  and  review  notes  or  other  course  material  after  the  face 
to  face  part  is  over. 

The  hybrid  course  design  was  different  than  WBI  in  that  it  combined  the  advantages  of  face  to  face  and 
online  modes  of  instruction.  Sands  (2002)  provided  proposals  for  hybrid  course  design  and  development.  Other 
studies  on  finding  the  ideal  hybrid  structure  were  done  by  (Marques  et.  ah,  1998)  and  Jones,  Cranitch  and  Jo 
(2001).  In  both  studies  hybrid  courses  were  developed  and  descriptive  studies  were  made.  Both  studies  found 
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the  hybrid  course  mode  superior  on  traditional  course  mode.  Studies  on  student  achievement  in  hybrid  course 
showed  that  students  were  more  successful  in  the  hybrid  courses  than  they  do  in  purely  web  based  or  traditional 
courses  (Lilja,  2001;  Truckman,  2002,  Christman  et.  ah, 1997;  Christman  and  Badget,  1999;  Persin  2002).  The 
literature  showed  that  students’  course  satisfaction  was  high  in  hybrid  courses  (Gray,  1999;  Black,  2002). 
Students’  attitudes  toward  technology  and  technology  integrated  courses  were  indicated  as  positive  in  hybrid 
courses.  Several  studies  showed  that  a “mixed”  course  structure  was  preferred  by  the  students  and  that  hybrid 
courses  effected  students  learning  positively  (Gunter,  2001;  Leon  de  la  Barra  et  ah,  1999). 

Student  Motivation 

There  are  many  factors  affecting  the  learning  in  hybrid  courses  and  one  of  the  important  factors  is  the 
source  of  motivation  of  students.  While  some  students  have  extrinsic  motivation  some  others  have  intrinsic 
motivation.  Extrinsic  motivation  in  hybrid  courses  can  lead  to  externally  rewarded  learning.  Examples  of  this 
extrinsic  motivation  are  grades,  time,  income,  legislative  power  and  so  on.  Intrinsic  motivation  can  lead  to 
internally  rewarded  learning.  Intrinsic  motivation  is  the  desired  motivation  type  in  courses  since  individual 
meaning- making  is  a critical  element  of  learning.  This  type  of  motivation  is  based  on  internal  values  like,  the 
will  to  learn,  the  desire  to  solve  a problem,  the  will  to  understand  the  course  content,  the  meaning  of  course 
content.  Intrinsic  motivation  can  lead  to  higher  levels  of  learning  and  critical  thinking  abilities. 

The  literature  shows  two  models  for  defining  motivation  of  students  for  learning.  The  first  one  is 
Malone’s  (Malone,  1981;  Malone  & Lepper,  1987,  both  cited  in  Alessi  and  Trollip,  2001,  p.25)  motivation 
theory  in  which  he  suggested  four  relevant  factors  of  motivation:  challenge,  curiosity,  control,  and  fantasy. 
Malone  and  Lepper  (1987  cited  in  Alessi  and  Trollip,  2001,  p.26)  identified  motivators  as  either  intrinsic  or 
extrinsic.  Extrinsic  motivators  were  described  as  independent  of  instruction.  Lepper’s  (1985,  cited  in  Alessi  and 
Trollip,  2001,  p.26)  research  provided  evidence  that  “extrinsic  motivators  diminish  one’s  interest  in  learning 
because  the  goal  becomes  the  reward  rather  then  their  learning”.  Malone  and  Lepper  (1987)  proposed  that 
intrinsic  motivators  play  a more  dominant  role  on  students’  learning  than  extrinsic  motivators. 

The  second  motivation  theory  was  that  of  Keller  (Keller  & Suzuki,  1988,  cited  in  Alessi  and  Trollip, 
2001,  p.25).  Similar  to  Malone’s  theory  he  also  suggested  four  components  as  essential  factors  of  student 
motivation:  attention,  relevance,  confidence,  and  satisfaction.  The  theory  is  known  as  Keller’s  ARCS  model  of 
motivation  design.  Keller  did  not  indicate  any  desirability  of  intrinsic  or  extrinsic  motivation,  but  rather  he 
argues  that  the  instructional  designer  must  be  proficient  at  motivation  design  as  well  as  instructional  strategy 
and  content  design. 


Method 

A “Computer  Networks  and  Communications”  course  was  offered  as  an  elective  course  to  all  students 
of  a state  university  in  Turkey  in  Fall  2002  semester.  Twenty-five  students  attended  the  hybrid  course.  In  the 
beginning  of  the  first  meeting  of  the  course  a short  orientation  about  how  to  use  the  web-site  was  given  to  the 
participants.  Students  were  informed  about  things  that  were  expected  from  them  while  using  the  web-site,  what 
the  security  policies  were,  how  the  site  functions,  what  the  Internet  address  of  the  web-site  was,  and  how  to 
choose  their  username  and  passwords.  Students’  web-site  usage  was  logged  by  the  log  system  and  the  durations 
and  activities  of  each  student  were  checked  each  week.  Every  student  had  to  visit  the  web-site  of  the  course  and 
had  to  be  active  for  at  least  one  hour  each  week.  The  student  could  not  leave  the  page  open  and  leave,  since  the 
system  logged  them  out  after  5 minute  inactive  time.  Students  met  once  a week  for  one  hour  to  participate  in  the 
classroom  activities  and  no  lecturing  was  done  in  these  meetings.  As  a prerequisite  to  the  course  all  students, 
were  required  to  have  taken  a computer  literacy  course.  This  was  required  to  assure  that  all  students 
participating  in  the  study  had  the  basic  knowledge  level  about  computers 

The  Website  of  the  Course 

The  “Computer  Networks  and  Communications”  course  was  designed  as  a hybrid  course  which 
required  self-paced  learning  time  since  the  course  content  was  online,  creating  a significant  reduction  in 
classroom  lecture  time.  The  course  was  offered  to  the  students  as  a hybrid  course,  which  was  a mixture  of  face 
to  face  instruction  with  online  learning.  For  this  purpose,  a web-site  was  developed  to  serve  as  the  Web-based 
learning  environment.  The  web-site  of  the  course  was  developed  by  two  faculty  members  of  the  Computer 
Education  and  Instructional  Technology  Department  of  a Turkish  state  university.  Some  of  the  Internet 
technologies  used  in  developing  the  web-site  can  be  listed  as,  Active  Server  Pages  (ASP),  Microsoft  SQL 
Server  7.0,  Dynamic  HyperText  Markup  Language  (DHTML),  and  Cascading  Style  Sheets  (CSS).  The  course 
web-site  consisted  of  course  content,  syllabus,  announcements,  assignments,  forum  and  comments  parts.  In  the 
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web-site  there  were  also  some  cognitive  tools  to  support  student  learning  such  as  highlight,  notebook, 
bookmark,  search,  glossary  and  history. 

Before  the  study,  the  “Computer  Networks  and  Communications”  course  was  given  as  a must  course. 
Although  the  course  had  been  offered  for  a long  time  there  were  no  written  goals  and  objectives.  The  first  step 
of  redesign  was  analysis  of  the  data  about  the  course.  Informal  and  formal  data  of  students  who  already  took  the 
course  in  terms  of  student  feedback  and  grades  were  investigated.  Existing  knowledge  and  skills  of  the  students 
who  registered  for  the  course  were  also  investigated.  As  the  second  step,  the  desired  outcomes  of  the  course  in 
terms  of  goals  and  objectives  were  specified  and  specific  learning  objectives,  assessment  instruments,  exercises, 
and  topics  to  be  included  were  documented.  These  were  used  to  determine  the  content  and  visual  elements  of 
the  web-site  of  the  course.  While  there  were  new  content  and  visuals  created  by  the  instructor,  because  of 
internal  validity  concerns,  majority  of  the  visual  elements  and  the  content  were  adapted  to  be  used  from  a 
commercially  well-known  information  source  with  permission.  As  the  third  step,  the  graphical  user  interface  of 
the  web-site  was  designed.  As  the  last  step  of  creating  the  web-site,  the  content  and  the  visual  materials  were 
coded.  The  content  was  structured  in  the  web-site  according  to  the  syllabus,  which  was  organized  week  by 
week.  The  design  and  the  web-pages  were  ready  to  use  before  the  course  started.  As  implementation,  chapters 
were  published  for  student  access  week  by  week.  The  effectiveness  of  the  design  and  training  materials  were 
continuously  evaluated  through  students’  comments.  The  web-site  of  the  course  was  a dynamic  one,  working 
with  conjunction  to  a database. 

Because  the  course  had  high  technical  knowledge  base  and  a loaded  content,  more  procedural 
knowledge  and  skills,  and  had  students  with  limited  prior  knowledge  about  the  content,  the  web-site  was  relying 
on  guided  learning  and  the  activities  in  the  classroom  on  discovery.  The  instructional  design  of  the  hybrid 
course  was  a mixture  of  objectivist  and  constructivists  approaches.  The  web-site  included 
objectivist/instructivist  (directed  learning)  and  constructivist  elements.  The  objectivist  structure  in  terms  of 
content  presentation  structure  in  the  web-site  was  supported  with  constructivist  elements  especially  in 
classroom  meetings.  Group  works,  games,  discussions  and  projects  were  constructivist  elements  planned  to  go 
hand  in  hand  with  the  online  part  of  the  course. 

The  users  were  authenticated  with  username  and  password  to  access  the  web-site  of  the  course.  The 
username  supplied  in  authentication  initiated  the  log  system,  which  was  internally  bound  to  a database,  to  keep 
track  of  the  activities  of  the  students  while  going  trough  the  content  and  using  the  cognitive  tools.  The  screen 
design  of  the  web-site  separated  the  web  page  into  two  main  parts.  One  part  was  used  for  visual  and/or 
graphical  elements,  and  the  other  part  was  used  for  content  related  text  in  the  whole  content  screen  to  provide 
the  consistency.  A sample  content  page  is  supplied  in  figure  1.  A “Jump  to”  tool  in  the  form  of  a drop-down 
menu  enabled  the  students  to  navigate  to  any  part  of  the  web-site  whenever  they  wanted.  The  students  could 
always  see  where  they  were  by  the  “You  are  here:”  tool.  Other  tools  to  mention  were  site  map  and  help,  which 
were  useful  for  students  in  navigating  between  different  parts  of  the  web-site. 

The  main  page  of  the  web-site  included  six  links  that  the  students  could  choose  from  (Course  Content, 
Syllabus,  Announcements,  Assignment,  Forum,  and  Comments).  Additionally,  there  were  three  message  notes, 
first  one  was  a message  from  the  instructor,  second  one  was  a note  written  by  the  students  themselves  to  remind 
them  things,  and  the  last  one  highlighted  the  last  content  the  current  user/student  visited. 


267 


3 Data  ErKAfxulatMm  (pogrb)  Mttfocoft  Intranet  l Kplnrrr 


• ’ J E)  (2  ilfgionto  Jrtaerf  Pj*  ® 


JSJil 


CEIT  314  You  ora  hare:  he  OS]  Model  > Data  Encapsulation  (pageo) 


Data  Encapsuladon 

PCU 


At  3 tiansmittng  device,  the  data 
encapsulation  method  worts  as  follows: 


^ 5v*yn,*"t 
^ P*X*  1 


^ 0‘ta 


1.  User  information  is  converted  to  data 
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Figure  1 - The  Content  Sturucture  of  the  Web-Site 


Procedures 

Every  student  in  the  hybrid  course  had  to  visit  the  web-site  of  the  course  and  had  to  be  active  for  at 
least  one  hour  each  week.  The  student  could  not  leave  the  page  open  and  leave,  since  the  system  logged  them 
out  after  5 minute  inactive  time.  In  the  one  hour  classroom  meeting,  students  with  suspicious  activities  and 
students  with  visiting  time  less  than  one  hour  were  informed  to  be  cautious  about  their  performance  related  with 
the  course  web-site.  The  differences  in  learning  and  teaching  activities  between  the  hybrid  course  and  the 
traditional  course  were  shown  by  using  Caladine’s  (1999)  model  which  he  called  “A  Model  for  Learning  and 
Teaching  Activities”  (MOLTA).  The  differences  between  the  two  courses  are  summarized  in  Table  1.  The 
common  activities  of  the  two  courses  are  shown  in  Figure  1.  MOLTA  classified  teaching  and  learning  activities 
into  five  elements;  delivery  of  material,  interaction  with  materials,  interaction  with  the  teacher,  interaction 
among  students  and  intra -action. 
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Table  1 -The  Differences  between  the  Hybrid  Course  and  the  Traditional  Course 


Element 

Traditional  Course 

(3  hours  of  classroom  meeting 
each  week) 

Hybrid  Course 

(1  hour  of  classroom  meeting 
each  week) 

Delivery  of  Material 

Lectures  supported  with 
PowerPoint  presentation 

Web-site,  on-line  materials 

Interaction  with  materials 

Text  books,  notes,  library 
books,  homework,  quizzes, 
classroom  activities. 

Multimedia,  web  browsing, 
cognitive  web  tools, 
homework,  quizzes,  classroom 
activities. 

Interaction  with  the  teacher 

Classroom  discussion,  face  to 
face  questions,  consultation 

Web  announcements,  forum, 
phone,  face  to  face  questions, 
consultation 

Interaction  between 
students 

Group  works,  classroom 
discussions,  projects, 
classroom  games 

Web  forum,  e-mail,  group 
works,  classroom  discussions, 
projects 

Intra-action 

Classroom  discussions,  group 
works 

Classroom  discussions,  group 
works,  web  forum 

The  students  in  the  hybrid  course  were  interviewed  individually  to  get  their  perceptions  about  the 
dimensions  of  the  hybrid  course  in  terms  of  their  effect  on  their  motivation.  The  students  in  the  hybrid  course 
were  interviewed  one-on-one  during  the  last  two  weeks  of  the  semester.  Each  interview  lasted  for  about  40-60 
minutes.  The  interviews  were  recorded  having  taken  students’  consent.  The  recorded  interview  data  was 
transcribed  and  analyzed  to  find  out  the  students’  motivation  types.  To  understand  students’  major  sources  of 
motivation,  content  analysis  was  carried  out  on  the  answers  to  the  questions  on  each  dimension. 

Results 

The  findings  of  the  study  showed  that  motivation  and  rewarded  learning  is  very  important  for  students’ 
learning  in  the  hybrid  course.  The  analysis  of  the  interview  data  to  find  out  the  type  of  motivation  that  was  more 
effective  on  students  learning  in  the  course  showed  that  students  had  both  type  of  motivation,  intrinsic  and 
extrinsic  but  one  of  them  was  more  dominant.  One  indication  for  intrinsic  motivation  was  “enjoying”  the 
course.  Students  indicated  that  they  enjoyed  some  learning  activities.  Students  did  not  enjoy  reading  the  content 
from  the  website,  but  they  enjoyed  the  real-life  experiences,  like  making  a cable  installation,  configuring  a 
computer  or  a network  device,  and  making  a cabling  design  for  a given  building  floor  plan.  They  also  enjoyed 
reading  and  applying  real  network  protocols  and  addressing  schemes  like  IP.  A student  said:  “I  always 
wondered  why  we  configured  the  computers  with  IP  address  and  subnet  mask.  Now  I understand  why  and  how 
we  use  it.”  There  were  parallel  comments  regarding  student  motivation  and  metacognition. 

Students  indicating  their  “joy”  of  learning  the  topics  in  the  course  were  those  students  with 
metacognitive  abilities  knowing  “what  they  learned”  and  “why  and  how  they  learned.”  For  example,  such  a 
student  said:  “I  expected  that  this  course  would  change  my  way  of  understanding  computer  networks  topic.  My 
expectation  became  true,  now  I look  at  many  things  different.  For  example,  when  I enter  a student  computer  lab 
I can  determine  that  the  line  is  going  from  there,  the  switch  is  located  there,  this  is  a good  or  bad  way  of 
installation.”  Students  were  asked  which  features  of  the  hybrid  course  they  liked  the  most  and  they  indicated  the 
following  features: 

1.  The  content  of  the  hybrid  course  (22  students ):  Computer  networks  subjects  were  found  interesting  by  most 
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of  the  students.  Students  stated  that  they  liked  to  learn  about  these  subjects  because  they  would  be  useful  in  their 
professional  life.  Almost  all  students  said  that  they  would  benefit  from  the  course  content  in  the  future. 


2.  The  hybrid  structure  of  the  course  (16  students):  Students  indicated  their  enjoyment  in  taking  an 
alternatively  delivered  course  after  so  many  traditional  courses.  It  was  something  new  for  them.  They  stated  that 
they  found  the  course  structure  interesting  and  useful.  They  especially  liked  the  course  not  being  fully  web 
based  or  fully  traditional. 

3.  The  learning/instruction  activities  done  in  the  classroom  (15  students ):  Students  stated  that  they  prefer 
doing  activities  rather  than  sitting  silently  and  listening  to  the  instructor.  They  indicated  that  they  have  enjoyed 
to  do  practice  on  the  information  they  read  from  the  web-site. 

4.  The  cognitive  tools  in  the  course  website  (14  students ):  According  to  student  comments,  the  cognitive  tools 
were  giving  the  course  web-site  a professional  feeling,  making  it  different  than  standard,  electronic  page  turning 
web-sites.  One  student  commented  on  this:  “The  tools  in  the  web-site  were  very  usable.  I used  them  for 
accessing  to  information  quickly  and  easily.” 

5.  The  web-site  of  the  course  (12  students ):  The  web-site  was  found  to  be  very  user-friendly,  nice  looking  in 
terms  of  graphics  and  well  organized  in  terms  of  access  to  information.  The  students  liked  the  navigation 
structure  and  the  information  presentation  structure. 

When  the  students’  interview  results  on  their  likes  and  dislikes  are  compared,  it  could  be  seen  that  the 
students  had  internal  and  external  motives  throughout  the  course.  The  new  hybrid  structure,  the  user-friendly 
structure  of  the  web-site  and  the  cognitive  tools  were  adding  to  students’  external  motivation.  Students’ 
enjoyment  of  the  classroom  activities  and  their  interest  in  learning  the  technology  related  to  computer  networks 
were  internal  motivation  in  the  hybrid  course.  One  common  view  of  students  was  that  the  classroom  meetings 
and  the  face  to  face  communication  with  instructor  and  the  peers  was  a source  of  motivation.  Students  indicated 
that  they  liked  especially  to  see  the  instructor  and  they  got  motivated  through  this.  Regarding  this,  while  some 
students  said  that  they  understood  the  topics  better  through  interaction  with  the  teacher  and  their  friends,  others 
indicated  they  liked  to  “talk”  with  the  others.  Detailed  analysis  pointed  towards  intrinsic  motivation  as  the  key 
element  for  success  in  the  hybrid  course.  The  findings  about  the  factors  effecting  the  students’  motivation  are 
summarized  in  Table  2. 
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Table  2-Summary  of  students’  motivation  types  in  the  hybrid  course 

Factors  effecting 
students  motivation 

Type  of  Motivation 

Effect  on  Students 

Studying  to  the  course 

Extrinsic 

Negative  - complained  about  health 

content  through  the  web- 

was an  obligation  to  read  at  least  for 

problems  like  eye  watering  and 

site 

one  hour, 

students  were  logged 

availability  of  internet  access 

The  structure  of  the 

Intrinsic 

Positive  - students  were  used  to 

course  in  terms  of 
“students  learning 
preferences” 

Individual  learning  was  supported 

individual  learning 

Negative-  students  expected  to  carry  on 
their  learning  habits 

The  structure  of  the 

Extrinsic 

Positive  - students  preferred  to  study  at 

course  in  terms  of 

to  choose  their  own  study  time 

their  homes,  get  access  to  course 

“logistic  preferences” 

class  hour  was  only  one  hour 

content  whenever  they  want. 

Expectations  from  the 

Extrinsic 

Negative  - Students  were  frustrated 

course  related  to  “external 

expecting  to  find  a job, 

easily  when  faced  with  the 

rewards” 

to  get  a certificate, 

dedicate  less  time  to  the  course 

requirements  of  the  course. 

Expectations  from  the 

Intrinsic 

Positive  - Students  liked  to  understand 

course  related  to  “internal 

Enjoying  learning  computer 

the  meaning  and  functioning  of 

reward” 

networks  related  topics. 

computer  networks  and  internet  they 
used  in  their  daily  life. 

Classroom  activities 

Intrinsic 

enjoying  to  do  practice  of  what  is  in 
the  course  content, 
being  active  rather  than  passive 
listeners 

Positive  - Students  could  use  and  show 
their  knowledge  to  their  peers  and  the 
instructor. 

Cognitive  tools  in  the 

Extrinsic  & Intrinsic 

Positive  - Students  could  customize  the 

course  web-site 

organizing,  searching  and  accessing 
information  fast  and  easily 

course  web-site  usage  according  to 
their  learning  preferences  like  taking 
notes,  highlighting  and  searching, 
bookmarking  and  so  forth. 

The  web-site  of  the  course 

Extrinsic 

appealing  in  terms  of  graphics 
design,  navigation  structure  and 
information  presentation 

Positive  - Students  found  the  web-site 
easy  to  study,  user-friendly  and  well 
organized. 

Course  content 

Intrinsic 
new  technology, 
subjects  are  valuable  in  the 
information  society 

Positive  - Students  were  aware  of  that 
they  learned  new  technologies. 
Negative  - Students  with  non  technical 
background  or  previous  knowledge 
found  the  subjects  too  technical  and 
hard  to  understand. 

The  instructor  of  the 

Intrinsic 

Positive  - Students  were  relaxed,  and 

hybrid  course 

student-teacher  interaction  was 
informal  and  friendly, 
teacher  motivated  students  with 
positive  feedback 

easily  communicated  with  each  other 
and  the  instructor  during  the  classroom 
activities. 
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Conclusion  and  Discussion 

Students’  answers  indicated  two  types  of  motivation  while  laming  in  a hybrid  course;  extrinsic  and 
intrinsic.  The  study  results  found  out  that  intrinsic  motivation  and  internally  rewarded  learning  is  the  key 
element  of  the  hybrid  course.  Interviews  revealed  that  students  with  extrinsic  motivation  are  more  prone  to 
losing  overall  motivation.  It  was  seen  that  some  students  in  the  hybrid  course  with  extrinsic  motivation  lost  their 
motivation  and  were  frustrated  by  the  course  content.  On  the  other  hand,  students  with  internal  motivation  were 
more  aware  of  objectives  of  the  course,  had  the  ability  to  plan  and  evaluate  their  own  learning.  They  also  had 
the  metacognitive  skills  which  are  referred  by  Flavell  (1979,  cited  in  Revees  and  Reeves,  1997)  as  skills  one  has 
in  learning  to  learn.  The  interview  results  also  indicated  that  the  source  of  motivation  is  not  discrete  but  a 
continuum.  This  can  be  interpreted  as  students  have  both  types  of  motivation  while  learning  but  intrinsic 
motivation  is  more  important  in  hybrid  environments. 

Research  points  on  motivation  as  an  important  factor  on  student  achievement.  There  is  also  research 
evidence  that  motivation  is  not  only  a determinant  for  student  achievement  but  it  has  to  be  activated  for  each 
task  (Weiner,  1990).  There  are  different  opinions  about  which  type  of  motivation  is  more  effective  on  students 
learning.  The  findings  of  the  current  study  points  towards  intrinsic  motivation  as  the  dominant  motivation  type 
in  students  learning  in  the  hybrid  course.  This  result  supports  the  findings  of  Lin  and  McKeachie  (1999,  cited  in 
Lee  & Park,  2003,  p.657)  who  suggested  that  intrinsically  motivated  students  engage  in  the  task  more 
intensively  and  show  better  performance  than  extrinsically  motivated  students.  However,  some  older  studies 
showed  opposite  results  for  traditional  classroom  settings  (Frase,  Patrick,  & Schumer,  1970,  cited  in  Lee  &Park, 
2003,  p.657).  The  contradictory  findings  have  been  explained  as  “possible  interaction  effects  of  different  types 
of  motivation  with  different  students.  For  example,  the  intrinsic  motivation  may  be  more  effective  for  students 
who  are  strongly  goal  oriented,  like  adult  learners,  while  extrinsic  motivation  may  be  better  for  students  who 
study  because  they  have  to,  like  many  young  children”  (Park  & Lee,  2003,  p.657). 
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Abstract 

Analyzing  data  from  the  Education  Longitudinal  Study  of 2002  (ELS:2002),  this  report  examines  how 
computer  use  produced  generic  benefit  to  all  children  and  differential  benefits  to  minority  and  poor  children. 
Specifically,  we  examined  computer  use  at  home  vis-a-vis  computer  use  at  school  in  relation  to  the  academic 
performance  of  disadvantaged  children  and  their  peers.  Home  computer  use  typifies  socially  differentiated 
opportunities,  whereas  school  computer  use  promises  generic  benefits  for  all  children. 

The  findings  suggest  that,  with  other  relevant  conditions  constant:  (a)  disadvantaged  children  did  not 
lag  far  behind  their  peers  in  computer  use  at  school,  but  they  were  much  less  likely  to  use  computers  at  home; 
(b)  computer  use  at  home  was  far  more  significant  than  computer  use  at  school  in  relation  to  high  academic 
performance;  (c)  using  a computer  at  school  seemed  to  have  dubious  effects  on  learning;  (d)  disadvantaged 
children  benefited  less  than  other  children  from  computer  use,  including  computer  use  at  home;  and  (e) 
compared  to  their  peers,  disadvantaged  children  's  academic  performance  seemed  less  predictable  by  computer 
use  than  other  predictor  variables. 

Introduction 

It  is  overly  simplistic  to  assume  that  new  technologies  applied  to  education  will  uniformly  benefit  all 
children  in  academic  achievement.  In  this  paper,  we  examine  the  relationship  between  computer  use  and 
academic  achievement  of  students  of  different  backgrounds.  We  propose  generic  benefits  versus  differential 
benefits  of  technologies  as  a conceptual  tool  to  understand  the  relationship  between  computer  use  and  student 
learning.  Assessing  how  computer  use  at  home  vis  -a-vis  computer  use  at  school  provide  generic  and  differential 
effects  on  student  achievement,  we  analyzed  data  from  the  Education  Longitudinal  Study  of  2002  (ELS  :2002). 
We  offer  brief  implications  for  improving  curricula  and  instruction  in  technology -related  programs  to  ensure 
equitable  education. 


Background:  Social  Stratification  of  Technologies 

Dramatic  technological  advances  promise  to  help  educators  realize  the  ideal  of  equal  educational 
opportunity.  Many  believe  that  with  powerful  and  cost-effective  technologies,  minority  and  poor  children  will 
be  able  to  receive  education  of  the  same  quality  as  their  more  fortunate  peers  (Gladieux  & Swail,  1999;  Panel 
on  Educational  Technology,  1997).  New  computing  and  network  technologies  can  provide  disadvantaged 
students  with  access  to  knowledge-building  and  communication  tools  and  more  individualized  learning 
opportunities. 

However,  access  to  technology  is  not  equitable  across  sociodemographic  categories  since  it  is 
determined  by  resources  available  to  the  schools,  communities,  and  households.  New  technologies  seem  to  best 
accommodate  those  who  already  take  advantage  of  available  educational  opportunities  (Barley,  1997).  It  is 
possible  that  use  of  these  technologies  may  widen  the  educational  gap  in  such  a way  that  “advantage  magnifies 
advantage”  (Gladieux  & Swail,  1999)  as  the  advantaged  benefit  most  from  cutting-edge  technologies  whereas 
the  most  needy  benefit  least.  Skeptics  question  whether  new  technologies  per  se,  are  able  to  improve 
educational  equity  since  both  access  to  and  use  of  technologies  are  socially  stratified. 

There  are  clear  patterns  of  uneven  distribution  of  access  to  technologies,  including  computer  and 
webTV  ownership,  Internet  access,  and  email  use  (U.S.  Department  of  Commerce,  1999).  To  date,  the  digital 
divide  issue  has  turned  on  the  concept  of  access  (Ba,  et  al,  2001).  Access  has  become  an  issue  of  social  equity. 
Equal  access  to  the  technology  and  the  skills  to  use  it  are  increasingly  necessary  for  economic  success  (Pachon, 
et  al,  2000).  Pearson  (2002)  indicated  that  there  are  large  disparities  between  the  access  opportunities  of  the  rich 
vs.  poor  and  ethnic  majority  vs.  ethnic  minority  populations.  Concerning  access  to  new  technologies,  poor  and 
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minority  students  are  at  a disadvantage. 

The  rates  of  the  Internet  access  among  individuals  with  high  income  and  higher  education  are  greater 
than  the  rates  among  those  with  low  income  and  less  education.  Race-ethnicity  was  an  important  stratification 
factor  in  the  rate  of  Internet  access.  Blacks  and  Hispanics  are  less  likely  to  have  Internet  access  at  home  than 
Whites  and  Asian  Pacific  Islanders,  although  the  gap  is  narrower  for  Internet  access  outside  the  home.  (U.S. 
Department  of  Commerce,  1999).  Uneven  availability  and  access  exist  among  public  schools  with  different 
socioeconomic  student  populations.  In  multiple  measures  of  access,  schools  with  a large  number  of  poor 
students,  receiving  free  or  reduced  price  lunch,  rated  lower  than  to  schools  with  smaller  numbers  of  poor 
students  (National  Center  for  Education  Statistics  [NCES],  1999b). 

Computers  have  been  increasingly  regarded  as  learning  tools  in  education,  but  not  a panacea  for 
educational  concerns  (Pachon,  2000).  However,  students  who  do  not  have  access  to  high-quality  computer 
experiences  at  home  or  school  are  not  being  provided  with  the  opportunities  they  need  to  be  successful  in 
society  (Pearson,  2002).  Lack  of  proper  education  is  an  important  barrier  to  technology  access  and  adoption 
(Hoffman  & Novak,  1999). 

The  process  of  using  technologies  is  socially  differentiated  as  well.  There  are  substantial  differences 
between  affluent  and  poor  schools  in  the  processes  used  by  teachers  in  instructing  their  students  on  computer 
and  Internet  use.  Teachers  and  students  in  poor  schools  are  more  likely  to  use  the  computer  for  drill  practice  and 
less  likely  to  use  it  for  research  work  when  compared  with  their  counterparts  in  affluent  schools  (NCES,  2000a). 
Disadvantaged  students  often  attend  unchallenging  computer-related  courses.  They  are  more  likely  to  take 
computer  literacy  classes  than  to  use  computers  in  the  study  of  key  subject  areas.  High-socioeconomic  status 
(SES)  students  are  more  likely  than  low-SES  students  to  engage  in  computer  programming  as  opposed  to  lower- 
level  computer-related  tasks)  and  to  use  computers  primarily  for  “higher-order”  or  “mixed”  activities  (rather 
than  drill-and-practice  activities).  For  challenging  computer  activities,  High-SES  students  disproportionately 
receive  better  learning  opportunities  than  poor  and  minority  students  (Wenlingsky,  1998).  School  reform 
involving  new  technological  applications  does  not  seem  to  narrow  the  divide,  as  revealed  in  a contrast  between 
an  impoverished  public  school  and  an  elite  private  school  (Warschauer,  2000).  Students  attending  different 
schools  are  systematically  channeled  into  distinctive  futures  via  the  process  of  assignment  to  technology-based 
programs:  for  the  affluent,  academic  and  research-oriented  higher  education;  for  the  poor,  workplace-oriented 
vocational  learning. 

Significantly,  access  to  technologies  at  home  has  a great  deal  to  do  with  how  technologies  are  learned 
in  school.  Students  whose  families  provide  ready  access  to  a computer  are  likely  to  take  advanced  computer 
classes  at  school  involving  such  tasks  as  the  analysis  of  complex  systems  and  college-oriented  academic  work. 
In  contrast,  students  who  have  no  experience  with  computer  at  home  often  are  placed  in  computer  courses 
emphasizing  routine  skill  learning  or  workplace -oriented  training  (e.g.,  Gladieux  & Swail,  1999;  Wenglinsky, 
1998). 


Concepts  and  Research  Issues 

We  used  a dual  construct  to  examine  computer  use  and  academic  performance,  namely,  technically 
generic  benefits  versus  socially  differentiated  benefits.  The  former  refers  to  the  possibility  that  application  of 
technology  consistently  benefits  every  student.  Socially  differential  benefits,  in  contrast,  hypothesize  that  the 
effects  of  technology  vary  by  the  social  grouping  of  its  users  and  by  the  social  settings  of  its  use. 

Under  the  rubric  of  generic  benefits,  educational  applications  of  technologies  such  as  online  instruction 
and  interactive  systems  allow  all  learners  to  readily  access  vast  amounts  of  information  and  to  learn  in  an 
individualized  process  that  accommodates  their  unique  needs,  abilities,  and  learning  styles,  thus  helping  to 
reduce  learning  gaps  related  to  students’  social  backgrounds  (Panel  on  Educational  Technology,  1997). 

The  perspective  of  socially  differentiated  benefits  argues  that  disadvantaged  children  do  not  benefit  from 
technologies  as  much  as  other  children  (Wenglinsky,  1998).  Disadvantaged  children,  even  with  access  to  new 
technologies,  are  more  likely  to  use  them  forrote  learning  activities  rather  than  for  intellectually  demanding 
inquiries.  The  social  conditions  in  which  educational  technologies  are  implemented  and  used  may  influence  the 
technologies’  ability  to  narrow  or  widen  historical  disparities.  Research  has  found  that  the  traditional  patterns  of 
classroom  organization  might  be  impermeable  to  change,  even  with  the  wide  availability  of  computers  at  school 
(e.g.,  Warschauer,  2000). 

With  the  perspective  of  social  stratification,  the  extent  to  which  educational  technologies  improve 
student  learning  varies,  partly  depending  on  students’  socio -demographic  backgrounds.  From  this  perspective, 
home  access  to  cutting-edge  technologies  is  a key  indicator  of  learning  opportunity.  Research  has  found  that 
children  with  access  to  computers  and  the  Internet  at  home  are  more  confident  and  resourceful  in  using 
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computer-related  technologies  at  school.  Lack  of  access  at  home,  even  when  access  is  provided  at  school, 
handicaps  many  poor  and  minority  children  in  productively  using  computers.  Home  access  to  a computer  and 
the  Internet,  differentiated  by  SES,  may  be  a significant  source  of  educational  inequality  in  the  United  States 
(Gladieux  & Swail,  1999). 

Further,  we  argue  that  computer  use  at  home  may  help  shape  fundamentally  different  attitudes  about 
using  technologies  for  learning.  Children  from  families  with  adequate  material  and  cultural  resources  tend  to 
“grow  up  with”  cutting-edge  technologies.  They  often  are  interested  in  new  technological  developments  and 
intimately  relate  themselves  to  these  changes  by  developing  some  sort  of  self-identity  with  technological 
products.  Constantly  curious  about  evolving  high  tech  areas,  these  children  are  able  to  actively  take  advantage 
of  new  technologies  for  the  study  of  core  academic  subjects  as  well  as  for  entertainment. 

In  contrast,  children  from  deprived  home  environments  are  not  only  unfamiliar  with  the  novel  ways  of 
learning  with  new  technologies,  but  also  could  be  alienated  from  the  rapidly -changing  technologies  that  they 
have  to  deal  with  outside  of  home,  including  those  found  in  the  classroom.  Without  a technology-friendly  home 
environment  to  foster  their  confidence  and  interest  in  computer-based  learning,  their  attitude  toward 
technologies  could  be  indifferent  or  even  hostile.  In  this  study,  we  see  that  the  use  of  computers  at  home  typifies 
the  socially  differentiated  benefit  of  technologies  because  it  is  largely  determined  by  family  material  and 
cultural  resources.  School-provided  access  to  computer-based  learning,  on  the  other  hand,  is  presumably  a 
remedy  to  the  social  stratification  of  technologies.  It  is  expected  to  provide  generic  benefits  of  technology  to  all 
children  since  public  schooling  by  default  promises  equitable  education.  Examining  computer  use  at  home  and 
at  school  is  thus  the  focus  of  this  study. 

Academic  achievement  is  also  conditioned  by  many  other  factors.  School  resources,  instruction  and 
curriculum,  teacher  expectation,  and  individual  students’  motivation  to  learn,  are  widely  documented  factors 
that  influence  academic  performance.  To  isolate  technologies’  generic  versus  differential  effects,  we  analyzed 
these  factors  together  with  computer  use  at  home  and  at  school  in  accounting  for  student  academic  performance. 

Specifically,  we  attempt  to  address  the  following  issues: 

1.  To  what  extent  did  disadvantaged  students  lag  behind  other  students  in  computer  use  at  school  and 
computer  use  at  home? 

2.  Ceteris  paribus,  how  did  computer  use  at  home  and  school  relate  to  high  school  students’  academic 
achievement  (generic  benefits)? 

3.  Does  the  relationship  between  computer  use  and  academic  achievement  differ  across  racial-ethnic 
and  SES  subgroups  (differential  benefits)?  and 

4.  Did  computer  use  help  narrow  achievement  gaps  associated  with  income  and  race-ethnicity  among 
the  NELS  cohort?  (gap-reduction  effect)? 


Methods 

Data  Source 

The  Education  Longitudinal  Study  of  2002  (ELS:2002)  will  provide  trend  data  about  critical 
transitions  experienced  by  2002  base  year  10th  grade  students  as  they  proceed  through  high  school  and  into 
postsecondary  education  or  their  careers.  Base  year  ELS:2002  was  carried  out  in  the  spring  term  of  the  2001-02 
school  year  with  a national  probability  sample  of  752  public,  Catholic,  and  other  private  schools.  Data 
collection  methods  consisted  of  five  separate  questionnaires  (student,  parent,  teacher,  school  administrator,  and 
library  media  center),  two  achievement  tests  (assessments  in  reading  and  mathematics),  and  a school 
observation  form  (facilities  checklist).  Base  year  questionnaires  were  completedby  15,362  of  the  17,591 
selected  sophomores,  13,488  parents,  7,135  teachers,  743  principals,  and  718  librarians.  The  multilevel  focus  of 
ELS:2002  provides  researchers  with  a comprehensive  perspective  of  influences  on  the  student  including  home, 
school,  and  the  community.  This  perspective  is  essentially  unified,  the  basic  unit  of  analysis  is  the  student. 

Multiple  regression  analysis  was  used  to  examine  each  independent  variable’s  relationship  with 
academic  performance,  controlling  for  the  other  variables.  A series  of  initial  tests  were  run  to  explore  alternative 
equations  that  could  yield  reasonably  good  fit  with  the  data.  In  the  final  analysis,  a series  of  equations  were 
specified  to  assess  the  racial-ethnic  and  SES  gaps  in  achievement  in  connection  to  computer  access  and  other 
variables. 

Variables 

A description  of  each  variable  follows.  The  extracted  data  were  edited  and/or  re -scaled.  Student 
academic  performance  represented  by  the  composite  math/reading  standardized  test  score  at  10th  grade  was 
used  as  the  outcome  indicator  in  this  study.  The  composite  score  is  the  average  of  the  math  and  reading 
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standardized  scores,  re -standardized  to  a national  mean  of  50.0  and  standard  deviation  of  10.0. The  standardized 
T score  provides  a norm-referenced  measurement  of  achievement  relative  to  the  population.  Spring  2002  10th 
graders,  as  a whole.  Race-ethnicity  was  a seven-category  variable  for  (a)  American  Indian/ Alaska  Native,  non- 
Hispanic,  (b)  Asian,  Hawaii/Pacific  Islander  (API),  non-Hispanic,  (c)  Black  or  African  American,  non- 
Hispanic,  (d)  Hispanic,  no  race  specified,  (e)  Hispanic,  race  specified,  (f)  Multiracial,  non-Hispanic,  and  (g) 
White.  In  multiple  regression  analysis,  the  grouping  was  dichotomous,  one  for  American  Indian/ Alaska  Native, 
Black  or  African  American,  Hispanic,  and  Multiracial,  and  the  other  for  White  and  API.  Combining  White  and 
API  into  a group  was  based  on  the  established  fact  that  the  API  group  on  average  has  similar  computer  access 
and  academic  performance  as  whites  (see,  for  example,  U.S.  Department  of  Commerce,  1999;  NCES  1999b; 
Jencks  & Phillips,  1998).  SES  was  indicated  by  a composite  score  derived  from  parents’  educational  attainment, 
parents’  occupation,  and  household  income.  A derived  quartile  variable  was  used  to  define  low-income  students 
as  those  who  were  in  the  lowest  quartile  of  the  derived  SES. 

Computer  use  was  represented  by  a series  of  variables,  including  student  self-reported  home  computer 
use,  school  computer  use,  frequency  of  computer  use  at  home  and  school,  different  modes  of  computer  use, 
computer  use  in  English  and  math  courses,  and  computer  use  by  English  and  math  teachers  for  instruction.  To 
examine  the  potential  generic  and  differential  benefits  of  technology  access  in  connection  to  academic 
achievement,  we  attempted  to  sort  out  complex  relationships  between  a group  of  relevant  explanatory  factors 
and  academic  performance  which  follow. 

School  factors  included  school  socioeconomic  composition,  school  geographic  locale  (urban, 
suburban,  and  rural),  and  school  provision  of  computer-related  programs  and  facilities.  Instruction/curriculum 
and  teacher’s  expectation  indicated  by  students’  placement  of  advanced  placement  program  (versus  general  and 
vocational  programs).  English  and  math  teachers’  expectation  for  students’  future  education  was  viewed  as 
another  condition  leading  to  meaningful  use  of  technology  in  academic  growth.  Family  resource  and  support 
indicated  by  SES,  availability  of  a home  computer,  and  parent’s  expectation  for  the  child’s  education. 

Analysis 

Variables  were  analyzed  through  two -sample  “t”  test  statistical  procedures  and  multiple  regression 
procedures.  In  the  bivariate  analysis,  a large  number  of  variables  conceptually  relevant  to  academic 
achievement  and  computer  access  were  examined.  Based  on  descriptive  and  bivariate  analysis  multiple 
regression  analysis  was  conducted  to  examine  the  predictor  variables’  unique  and  joint  relationships  with 
academic  performance.  A series  of  initial  tests  were  run  to  explore  alternative  equations  that  could  yield 
reasonably  good  fit  with  the  data.  Particular  attention  was  paid  to  testing  of  two-way  interaction  effects  in  order 
to  detect  joint  effects  of  predictors  on  achievement.  The  tests  include  interactions  between  computer  use/access 
and  race -ethnicity,  SES,  curriculum  and  coursework,  teacher  educational  expectations,  and  parent  student 
educational  expectations. 

In  the  final  analysis,  a series  of  equations  were  specified  to  assess  the  racial-ethnic  and  SES  gaps  in 
computer  access  and  the  possible  generic  and  differential  benefits  of  computer  use  on  academic  performance. 
The  first  equation  simply  demonstrates  the  existing  racial-ethnic  and  SES  gaps  in  computer  access. 
Subsequently,  school,  program,  family,  and  psychobehavioral  variables  are  entered  into  the  equations  to 
estimate  how  the  two  gaps  might  change.  SPSS  vl2.0  was  used  to  conduct  descriptive  procedures  and  AM 
v.0.06,  provided  by  the  American  Institutes  of  Research  and  Jon  Cohen  and  recommended  by  NCES  for  use 
with  ELS:2002  data,  was  used  to  conduct  multiple  regression  procedures. 

Results 

Research  Issue  1:  Computer  Access  and  Use 

Differences  in  computer  use  at  home  are  evident  both  in  race/ethnicity  and  SES  subgroups  (see  Table 
1).  With  regard  to  race,  APIs  (41.58%)  and  Whites  (40.49%)  frequency  of  computer  use  a home  was  well  above 
the  frequency  of  use  for  minorities,  specifically  Blacks/ African  Americans  (28.40%)  and  Hispanics  (26.98%). 

As  one  might  expect,  the  low  SES  subgroup  revealed  less  frequency  of  computer  use  when  compared  to  higher 
SES  subgroups.  However,  as  we  alluded  to  earlier,  frequency  of  computer  use  at  school  was  relatively  similar 
across  all  race/ethnicity  and  SES  subgroups.  Frequency  of  computer  use  for  school  work  revealed  differences  in 
the  race/ethnicity  subgroups,  again  between  the  non-API  minorities  using  computers  less  and  Whites/ APIs  with 
a higher  percentage  of  computers  use.  The  low  SES  subgroup  revealed  less  computer  use  for  school  work  than 
higher  SES  subgroups.  Only  minor  differences  in  use  of  computers  by  students  to  learn  on  their  own  were 
evident  in  race/ethnicity  and  SES  subgroups. 


277 


Table  1 Computer  Access/Use  Gaps:  Percentage  of  Computer  Access/Use  by  Race-Ethnicity  and  Low-Income 
Status  Subgroups  of  Base  Year  2002  Sophomores 

How  often  uses  How  often  uses  How  often  uses  How  often  uses 

computer  at  computer  at  computer  for  computer  to 
Subgroup  home1*  schoolb  school  workc  learn  on  ownd 


Amer.  Indian/ Alaska  Native 

30.17% 

19.52% 

19.89% 

18.65% 

Asian,  Hawaii/Pac.  Islander 

41.58 

15.37 

30.94 

25.58 

Black  or  African  American 

28.40 

17.32 

21.21 

22.14 

Hispanic,  no  race  specified 

29.71 

15.21 

21.11 

20.31 

Hispanic,  race  specified 

26.98 

14.61 

21.08 

19.34 

Multiracial,  non-Hispanic 

34.29 

16.94 

22.80 

21.05 

White 

40.49 

16.62 

24.97 

21.07 

Other  SES 

39.74 

16.28 

25.03 

21.04 

Low  SES 

26.71 

16.93 

17.98 

17.16 

Note.  Values  enclosed  in  parentheses  represent  standard  errors.  From  U.S.  Department  of  Education,  National 
Center  for  Educational  Statistics,  Education  Longitudinal  Study  of  2002  (ELS:2002),  “Base  Year”  panel  data. 
“Based  on  BYS47A,  a scale  ranging  from  1 through  5 indicating  increasing  frequency  of  computer  use,  where 
use  was  defined  as  a 4 and  5 rating  average.  '’Based  on  BYS47A.  “Based  on  BYS47A.  dBased  on  BYS47A. 

Research  Issue  2:  The  Generic  Benefits  of  Computer  Use 

We  examined  the  generic  benefit  of  computer  use  at  home  and  at  school  with  different  variables  in 
relation  to  the  math  and  reading  composite  score,  upon  controlling  for  the  effects  of  variables  that  have  been 
documented  as  relevant  to  achievement  ( ceteris  paribus  for  correlation  statements  thereafter).  In  Table  2,  with 
the  first  equation,  we  estimated  the  achievement  gaps  associated  with  SES  and  race-ethnicity.  SES  is  a strong 
positive  predictor  of  the  achievement  (with  beta=  4.94,  and /?<0.01).  We  separately  estimated  the  racial 
differences  with  six  binary  variables,  each  representing  a contrast  between  a given  minority  group  and  Whites. 
The  API  and  Multiracial  groups  had  a higher  average  score  than  the  Whites  (beta=-4.68  and  -1.79  at/? <0.01 
respectively).  Blacks/African  Americans  and  Hispanics,  race  specified,  showed  significantly  lower  average 
achievement  (2.31  and  0.91  respectively,  at  the  /?<0.01  level).  The  American  Indian/ Alaska  Native  and 
Hispanic,  race  not  specified,  revealed  no  significance. 

We  then  recoded  the  race-ethnicity  into  a single  binary  variable,  which  contrasted  non-API  minority 
groups  with  Whites  and  APIs.  In  equation  2,  we  entered  a set  of  individual  and  school  background  variables  that 
were  presumably  predictive  of  achievement,  together  with  SES  and  the  non-API  minority  dichotomy.  This 
procedure  allowed  us  to  demonstrate  that  most  background  variables  were  related  to  achievement,  as  expected, 
and  then  to  further  test  the  effects  of  computer  use/access  measures  after  controlling  for  these  background 
variable  s . 

Note  that  the  achievement  gaps  related  to  SES  and  race -ethnicity  decreased  as  those  individual  and 
school  variables  entered  into  the  equation.  This  implies  that  those  predictor  variables  accounted  for  a large 
portion  of  the  two  gaps,  meaning  that  providing  the  similar  conditions  on  those  variables,  low-income  and 
minority  students  would  have  done  less  poorly  in  math  and  reading  tests  relative  to  Whites  and  APIs. 

To  identify  a generic  benefit  of  computer  use  and  access  in  raising  the  achievement  level,  we  entered 
into  equation  3 a group  of  nine  variables  measuring  computer  use  and  access.  Of  these  variables,  six  estimates 
were  statistically  significant.  Owning  a home  computer  was  found  to  be  significantly  related  to  high 
achievement  (beta=1.05  at  /><0.01).  Three  variables  related  to  frequency  of  computer  use  at  home,  at  school, 
and  computer  use  for  school  work  revealed  significance  (beta=0.62,  0.36,  and  0.96  respectively,  all  at  the 
/?<0.01  level).  While  computer  use  in  9th  grade  fall  and  spring  math  did  not  produce  significance,  computer  use 
in  9th  grade  fall  and  spring  English  revealed  an  interesting  finding:  both  produced  significance  at  the  /?<0.01 
level,  however,  the  effect  was  different  for  fall  and  spring  (beta=-1.25  and  1.45  respectively). 


Table  2 SES  and  Racial-Ethnic  Gaps  in  Math  and  Reading  Composite  Test  Score  and  Generic  Benefit  of  Access 
to  and  Using  Computer:  Multiple  Linear  Regression  Coefficient  Estimates 


Independent  Variables 

Equation  1:  SES 

Equation  2: 

Equation  3: 

and  race- 

SES  and  racial- 

Generic  benefit 

ethnicity  gaps 

ethnic  gaps  net 

of  computer 

of  backgrounds 

access/use 
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Socio-economic  status  composite,  v.2 

4.94  (0.14)** 

4.80  (0.14)** 

3.92  (0.17)** 

Non-API  minorities  vs.  White 

— 

5.34  (0.26)** 

4.42  (0.30)** 

Amer.  Indian/ Alaska  Native,  non-Hispanic  vs 
White 

0.68  (0.72) 

— — 

— 

Asian,  Hawaii/Pac.  Islander,non-Hispanic  vs. 
White 

-4.68  (0.39)** 

— 

— 

Black  or  African  American,  non-Hispanic  vs. 
White 

2.31  (0.29)** 

— 

— 

Hispanic,  no  race  specified  vs.  White 

0.28  (0.34) 

— 

— 

Hispanic,  race  specified  vs.  White 

0.91  (0.35)** 

— 

— 

Multiracial,  non-Hispanic  vs.  White 

-1.79  (0.42)** 

— 

— 

Advanced  Placement  Courses 

4.36  (0.25)** 

4.12(0.27)** 

School  urbanicity 

0.05  (0.21) 

0.01  (0.22) 

Grade  10  percent  free  lunch-categorical 

-0.52  (0.34) 

-0.44  (0.44) 

Family  has  a computer 

1.05  (0.51)* 

How  often  uses  computer  at  home 

0.62  (0.13)** 

How  often  uses  computer  at  school 

0.36  (0.12)** 

How  often  uses  computer  for  school  work 

0.96  (0.13)** 

How  often  uses  computer  to  learn  on  own 

0.08  (0.10) 

Used  computer  in  9th  grade  fall  English 

-1.25  (0.49)** 

Used  computer  in  9th  grade  spring  English 

1.45  (0.39)** 

Used  computer  in  9th  grade  fall  math 

-0.37  (0.48) 

Used  computer  in  9th  grade  spring  math 

-0.55  (0.46) 

Intercept 

73.40 

69.80 

63.22 

R2 

0.26** 

0.29** 

0.33** 

Number  of  parameters 

7 

5 

14 

N of  weighted  cases2 

15362 

11639 

8647 

Note.  Values  enclosed  in  parentheses  represent  standard  errors.  From  U.S.  Department  of  Education,  National 
Center  for  Educational  Statistics,  Education  Longitudinal  Study  of  2002  (ELS:2002),  “Base  Year”  panel  data. 
aThe  number  of  cases  changed  across  equations  due  to  list-wise  deletion  of  missing  cases. 

* p<0.05.  **p<0.01. 

Research  Issue  3:  The  Differential  Benefits 

Does  computer  use  help  some  children  but  not  others?  Or  does  it  help  one  group  more  than  other 
groups?  To  examine  the  role  of  computer  use  in  promoting  academic  performance  of  students  of  different  SES 
and  racial-ethnic  backgrounds,  we  separated  the  analysis  by  the  subgroups.  Table  3 shows  multiple  regression 
coefficient  estimates  for  comparison  of  non-Asian  minorities  against  APIs  and  Whites  and  of  the  low-SES 
group,  defined  by  the  lowest  quartile  of  the  SES  composite  score,  against  the  group  of  other  SES  quartiles. 
Between  the  two  racial-ethnic  groups,  there  were  differences  in  effects  of  several  predictor  variables  including 
advanced  placement  courses,  and  parents’  expectations  for  students’  college  education. 

One  particular  computer-relevant  variable  differed  in  relation  to  achievement  across  the  both  race  and 
SES  groups.  Computer  use  for  school  work  produced  a positive  effect  on  the  API  and  White  group  (beta=  0.39 
and  p<0.01)  whereas  it  did  not  make  a difference  among  minority  students.  This  variable  also  produced  a 
positive  effect  on  the  Other  SES  quartile  group  (beta=  0.33  and  p<0.01)  whereas  it  did  not  make  a difference 
among  Low  SES  quartile  students. 


Table  3 Examining  Differential  Benefit  of  Access  to  and  Using  Computer  by  Race -Ethnicity  and  SES:  Multiple 
Linear  Regression  Estimates  for  Racial-Ethnic  and  SES  Subgroups 

Independent  variables  Non-API  API  and  White  Lowest  SES  Other  SES 

minority  students  quartile  quartile 

students  students  students 


Socio-economic  status  composite,  v.2 
Non-API  minorities 

Grade  10  percent  free  lunch-categorical 
School  urbanicity 
Advanced  Placement  Courses 


2.83  (0.21)** 

-0.10(0.05)* 
0.45  (0.30) 
2.51  (0.40)** 


2.97  (0.18)** 

-0.05  (0.04) 
0.10(0.23) 
3.50(0.27)** 


4.84(0.40)** 

-0.05(0.05) 

0.19(0.28) 

2.57(0.55)** 


5.72  (0.29)** 
-0.11  (0.04)** 
-0.01  (0.22) 
3.53  (0.26)** 
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0.22(0.04)** 


0.24(0.06)** 


0.25  (0.04)** 


How  far  teacher  expects  student  to  get  in  0.21  (0.05)** 
school  (English) 


How  far  teacher  expects  student  to  get  in 
school  (math) 

0.21  (0.05)** 

0.39(0.04)** 

0.29(0.05)** 

0.38  (0.04)** 

How  far  in  school  student  thinks  will  get- 
composite 

0.62  (0.07)** 

0.54(0.05)** 

0.52(0.08)** 

0.69  (0.05)** 

How  far  in  school  parent  wants  10th 
grader  to  go-composite 

0.63  (0.11)** 

1.39(0.09)** 

1.06(0.11)** 

1.33  (0.09)** 

Family  has  a computer 

0.27  (0.04)** 

0.18(0.04)** 

0.28(0.05)** 

0.20  (0.04)** 

How  often  uses  computer  at  home 

0.34  (0.07)** 

0.26(0.07)** 

0.25(0.08)** 

0.40  (0.07)** 

How  often  uses  computer  at  school 

0.01  (0.08) 

0.04(0.06) 

0.01(0.09) 

-0.04  (0.06) 

How  often  uses  computer  for  school 
work 

-0.09  (0.10) 

0.39(0.08)** 

0.02(0.11) 

0.33  (0.09)** 

How  often  uses  computer  to  learn  on  own 

0.09  (0.09) 

-0.01(0.07) 

0.00(0.10) 

0.01  (0.07) 

Mean  Square  Error 

64.02 

58.85 

59.92 

63.76 

R2 

0.23** 

0.33* 

* 

0.24** 

0.32** 

Number  of  parameters 

13 

13 

13 

13 

N of  weighted  cases 

4788 

9689 

2945 

10146 

Note.  Values  enclosed  in  parentheses  represent  standard  errors.  From  U.S.  Department  of  Education,  National 
Center  for  Educational  Statistics,  Education  Longitudinal  Study  of  2002  (ELS:2002),  “Base  Year”  panel  data. 

* p<0.05.  **p<0.01. 

Research  Issue  4:  The  Gap-Reduction  Effect 

How  does  computer  use/access  at  home  help  narrow  achievement  gaps  associated  with  income  and 
race-ethnicity?  With  frequency  of  home  computer  we  could  distinguish  the  ELS:2002  respondents  into  three 
groups.  A majority  group  (n=5,667)  included  the  students  who  use  computers  at  home  ranging  from  once  or 
twice  a week  to  everyday  or  almost  everyday  (frequent),  the  second  group  (n=  1,061)  who  use  computers  at 
home  ranging  from  less  than  once  a week  to  never  (not  frequent),  and  a third  group  (673)  that  did  not  have  a 
computer  at  home  (no  computer)(see  Table  4).  Separately  estimating  the  same  regression  equation  for  the  three 
groups  revealed  considerable  differences  in  academic  achievement  gaps  relating  to  income,  race-ethnicity,  and 
other  relevant  variables.  The  advanced  placement  course  variable  produced  a positive  effect  on  the  frequent 
home  computer  use  group  (beta=2.64  at  pc. 01)  with  no  effect  on  the  not  frequent  and  no  home  computer 
groups.  Variables  related  to  college  expectations  including  English  and  math  teachers’,  parents’,  and  student’s, 
all  produced  a positive  effect  on  students  that  own  a home  computer.  Frequency  of  computer  use  at  school 
revealed  positive  effects  for  the  frequent  and  not  frequent  home  computer  use  groups  (beta  0.31  at  p<0.01  and 
beta  0.70  at  pc. 05  respectively).  The  effect  of  the  frequency  of  computer  use  to  learn  on  own  variable  revealed  a 
positive  effect  (beta  0.32  at  pcO.Ol)  for  the  frequent  home  computer  use  group  and  a negative  effect  (-0.61  at 
pc0.05)  for  the  group  with  no  home  computer. 

Table  4 Gap-Reduction  Effect:  Multiple  Linear  Regression  Coefficient  Estimates  in  Equations  for  Students  Who 
Used  PC  at  Home  and  Students  Who  Did  Not  Have  a PC 

Students  that  used 
PC  at  home  once  or 

twice  a week  to  Students  that  used  Students  that  did 

everyday  or  almost  PC  at  home  less  than  not  have  a PC  at 

Independent  variables  everyday  once  a week  to  never  home 

Socio-economic  status  composite,  v. 2 1.41  (0.20)**  1.02  (0.40)*  0.39  (0.57) 

Non-API  minorities  3.85  (0.36)**  4.61  (0.60)**  4.03  (0.66)** 

Grade  10  percent  free  lunch-categorical  -0.45  (0.08)**  -0.51  (0.18)**  -0.66  (0.20)** 

School  urbanicity  0.57  (0.22)**  0.44  (0.40)  0.23  (0.46) 

Advanced  Placement  Courses  2.64  (0.26)**  0.94  (0.72)  0.09  (0.92) 

How  far  teacher  expects  student  to  get  in  1.71  (0.12)**  1.50  (0.24)**  1.56  (0.28) 

school  (English) 

How  far  teacher  expects  student  to  get  in  2.02  (0.11)**  1.87  (0.23)**  2.00  (0.29) 

school  (math) 
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How  far  in  school  student  thinks  will 

0.11 

(0.06)** 

0.29 

(0.13)* 

0.14 

(0.14) 

get-composite 

How  far  in  school  parent  wants  10th 
grader  to  go-composite 

0.54 

(0.10)** 

0.67 

(0.19)** 

0.40 

(0.23) 

How  often  uses  computer  at  school 

0.31 

(0.12)** 

0.70 

(0.29)* 

0.63 

(0.34) 

How  often  uses  computer  for  school 
work 

-0.08 

(0.15) 

-0.15 

(0.28) 

0.01 

(0.32) 

How  often  uses  computer  to  learn  on 

0.32 

(0.10)** 

0.21 

(0.26) 

-0.61 

(0.29)= 

own 

Mean  Square  Error 

45.18 

50.73 

50.26 

R2 

0.50** 

0.44** 

0.38** 

Number  of  parameters 

12 

12 

12 

N of  weighted  cases 

5667 

1061 

673 

Note.  Values  enclosed  in  parentheses  represent  standard  errors.  From  U.S.  Department  of  Education,  National 
Center  for  Educational  Statistics,  Education  Longitudinal  Study  of  2002  (ELS:2002),  “Base  Year”  panel  data. 
* p<0.05.  **p<0.01. 


Discussion 

Our  analysis  of  the  NELS  data,  adjusted  for  a series  of  individual  and  school  background  factors, 
generated  the  following  findings:  (a)  disadvantaged  children  did  not  lag  far  behind  their  peers  in  computer  use 
at  school,  but  they  were  much  less  likely  to  use  computers  at  home;  (b)  computer  use  at  home  was  far  more 
significant  than  computer  use  at  school  in  relation  to  high  academic  performance;  (c)  using  a computer  at  school 
seemed  to  have  dubious  effects  on  learning— taking  computer  science  courses  at  school  related  consistently  to 
low  performance  for  both  the  disadvantaged  and  their  peers;  (d)  disadvantaged  children  benefited  less  than 
other  children  from  computer  use,  including  computer  use  at  home;  and  (e)  compared  to  their  peers, 
disadvantaged  children’s  academic  performance  seemed  less  predictable  by  computer  use  than  other  predictor 
variables. 

Income  is  a stronger  indicator  than  race  regarding  the  use  of  computers  and  students’  achievement,  and 
the  strength  of  the  evidence  seems  to  be  clear  that  socioeconomic  factors  appear  to  play  a disturbing  role  in 
student  access  to  computers.  In  many  cases,  there  are  demographic  correlations  between  ethnicity  and  income 
level;  however,  affluence  is  the  key  factor  in  determining  the  positive  influence  of  computer  use  on  student 
performance.  Focus  should  therefore  be  given  not  only  to  racial  minorities  but  also  to  the  SES  minority  in  order 
to  best  implement  technology  for  achievement. 

Computer  use  at  home  was  far  more  significant  than  computer  use  at  school  in  relating  to  high 
academic  performance,  but  this  effect  was  absent  for  minority  and  low-SES  children.  These  findings  support  the 
notion  that  seemingly  ubiquitous  computer-based  technologies  are  nevertheless  differentially  available  and 
functioning  by  social  and  demographic  groups.  Public  education  has  not  remedied  the  problems  imposed  by  the 
social  stratification  of  technologies.  The  findings  refute  the  over-simplistic  belief  that  application  of  technology 
could  benefit  all  children  in  public  schools  by  closing  achievement  gaps. 

This  analysis  seems  to  underscore  a need  for  reform  of  technology  policies  and  computer-related 
curricula/instruction  to  provide  equitable  education  for  all  children.  The  pattern  that  computer  science  classes  in 
general  were  related  to  low  achievement  points  to  the  possibility  that  ill-designed  curriculum  or  poor  instruction 
rendered  such  technology-oriented  programs  disappointing.  Also,  achievement-irrelevance  of  a number  of 
variables  of  computer  use-at  school  or  setting-free-suggests  that  technologies  per  se  may  not  work  to  help 
performance.  Especially,  technologies  alone  would  not  work  well  for  closing  achievement  gaps  as  the 
performance  of  minority  and  poor  children  was  related  to  computer  use  to  only  a limited  extent. 

These  findings  present  clear  evidence  in  terms  of  the  relationship  between  socioeconomic  factors, 
equitable  distribution  and  use  of  computers,  teacher  technology  training,  and  students’  performance.  In  light  of 
this,  it  is  imperative  that  “equity”  in  school  computer  usage  must  involve  not  only  equity  in  access  but  also 
equity  in  consideration  of  the  learning  needs  of  low-income  and  minority  students.  It  follows,  then,  that  teacher 
technology  training  is  as  important  as  socioeconomic  factors  in  determining  the  level  of  SES  achievement  by 
the  career  graduate.  Increased  access  to  computers  will  only  have  positive  results  when  the  educator  has  a 
complete  grasp  of  the  role  and  use  of  computers,  and  an  understanding  of  the  student’s  home  environment  and 
how  their  deficiencies  must  be  met  in  order  to  realize  their  full  potential,  thus  enhancing  society  instead  of 
reducing  the  average  achievement. 
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Summary 

This  report  describes  an  evaluation  of  Computer  Literacy,  which  is  an  undergraduate  general  studies 
course,  offered  by  College  of  Education  at  a large  southwestern  university.  The  purpose  of  this  course  is  to 
provide  knowledge  about  computer  and  computing,  and  application  skills  in  using  Microsoft  Office  software. 
The  course  is  coordinated  by  a professor  and  is  currently  delivered  in  multiple  sections  by  teaching  assistants. 

The  evaluation  employed  a questionnaire  survey  and  an  interview  as  the  primary  data  collection 
methods.  In  total,  329  students  and  all  11  instructors  responded  to  the  survey.  The  interview  data  were  collected 
from  five  student  focus  groups,  five  instructors,  and  the  course  coordinator.  In  addition,  mean  scores  of  quizzes, 
the  midterm  exam  and  the  final  were  obtained  from  all  class  sections. 

The  survey  and  interview  results  suggested  that  the  topics  taught  were  useful  to  students,  especially  the 
hands-on  skills.  Among  the  different  teaching  strategies  used,  hands-on  projects  and  in-class  activities  were 
found  to  be  more  helpful  to  students,  whereas  reading  from  textbook  and  online  discussion  forum  were  not 
helpful.  The  results  suggest  that  ‘learning  by  doing’  is  good  for  learning  skills,  whereas  reading  and  group  work 
are  not  helpful  unless  they  can  make  the  class  more  interesting. 

Overall,  the  evaluation  revealed  that  the  Computer  Literacy  course  is  successful  in  providing  basic 
understanding  and  application  of  computer.  Recommendations  on  alternative  teaching  strategies  are  made  to 
improve  learning  effectiveness. 


Introduction 

Computer  literacy  is  expected  for  both  academic  and  career  achievement  (Davis,  1999).  Instructors  at 
school  expect  their  students  to  have  some  degree  of  computer  literacy  when  they  enter  college  (Hirschbuhl  & 
Faseyitan,  1994)  and  when  they  graduate  (Furst-Bowe,  Boger  & Franklin,  1995).  In  the  job  market,  corporate 
recruiters  have  reported  that  core  computer  competency  skills  are  very  important  to  the  employability  of  a 
recent  college  graduate  (Davis,  1997).  Ndahi  and  Gupta  (2000)  explained  that  this  is  because  computer  literate 
employees  have  better  prospective  in  workplace-specific  training  and  more  likely  to  be  successful  in  their  field. 
As  a result,  employers  seek  computer  literacy  in  almost  everyone  their  hire  (Ndahi  & Gupta,  2000).  Thus,  a job 
applicant  who  lacks  computer  skills  is  seriously  handicapped,  both  in  obtaining  a job  and  qualifying  for 
promotion  (Martin,  Carrier  & Hill,  1997). 

To  meet  the  demand  of  computer-literate  graduates,  it  is  important  to  determine  what  constitutes 
desired  computer  competency  and  how  they  should  be  taught.  Tucker  and  Garnick  ( 1991)  argued  that  a 
computer  literacy  course  should  be  characterized  to  emphasize  using  computers  as  tools  alongside  the  uses  of 
computers  in  society.  Ndahi  and  Gupta  (2000)  conducted  a survey  on  computer  literacy  in  workplace  training, 
and  the  results  suggested  that  word  processing  was  the  most  required  skill.  The  report  also  showed  a strong 
interest  in  learning  file  management  and  making  a PowerPoint  presentation.  The  demand  for  knowing  database 
software  was  also  increasing.  These  results  were  consistent  with  Furst-Bowe,  Boger  and  Franklin’s  study 
(1995),  which  indicated  that  skills  in  word  processing,  spreadsheets,  database  management,  graphics,  and 
information  retrieval  are  required  for  many  jobs.  In  terms  of  teaching  strategies,  previous  studies  found  that 
students  learn  more  when  they  are  allowed  to  work  in  a lab  together,  or  when  informal  peer  tutoring  and  support 
are  available  (Davis,  1999).  Learners  also  prefer  to  customize  learning  at  their  own  pace  (Smith  & Tarkow, 
1998). 

Course  description 

The  evaluated  Computer  Literacy  course  is  a 3 -credit  general  studies  course  required  for  undergraduate 
students.  In  the  14  three-hour  weekly  meetings  with  instructors,  students  learn  not  only  fundamental  concepts 
and  knowledge  about  computers,  such  as  how  data  are  input,  processed,  stored  and  output  (IPSO),  but  also  how 
to  actually  use  computer  applications  for  productivity,  such  as  using  Microsoft  Word,  Excel  and  PowerPoint. 
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The  administering  of  the  course  incorporates  Blackboard,  a course  management  system  widely  used  on  campus. 
All  course  information,  including  presentation  notes,  hand-outs,  assignments,  and  quizzes/exams  can  be 
accessed  from  Blackboard. 

In  spring  2003,  there  were  444  students  enrolled  in  21  sections  of  the  Computer  Literacy  course  taught 
by  1 1 teaching  assistants.  The  class  size  of  each  section  ranged  from  18  to  24  people.  One  professor  coordinated 
all  these  sections.  A formal  evaluation  was  conducted  to  improve  learning  effectiveness. 

Elements  evaluated  include  the  contents  taught,  the  skills  learned,  and  the  teaching  strategies  used  in 
the  course.  Student  and  instructor  perceptions  of  the  course  were  also  noted. 

Method 

Participants 

The  participants  in  the  evaluation  were  the  students,  instructors,  and  coordinator  of  the  Computer 
Literacy  course.  An  online  survey  was  administered  to  the  444  undergraduate  college  students  enrolled  in  the 
course.  Twenty-five  students  were  interviewed  in  focus  groups.  They  were  volunteers  from  five  different 
sections  with  five  students  in  each  group.  The  1 1 instructors  who  taught  this  course  were  surveyed  and  five  of 
them  were  interviewed.  The  course  coordinator  was  also  interviewed. 

Data  sources 

A variety  of  evaluation  instruments  were  employed  to  assess  the  course  in  a comprehensive  manner. 
Data  were  collected  with  regards  to  the  contents  covered,  skills  learned,  teaching  strategies  used,  and  student 
and  instructor  attitude  towards  the  course.  The  instruments  include:  1)  Student  survey,  2)  Instructor  survey,  3) 
Student  interview  of  the  focus  groups,  4)  Interview  of  instructors,  5)  Interview  of  the  course  coordinator,  and  6) 
Test  scores  of  all  21  sections. 

Student  survey  . The  26-item  student  survey  was  designed  to  determine  the  general  perception  of  the 
Computer  Literacy  course  in  terms  of  the  contents  covered  and  teaching  strategies  used.  The  survey  was 
distributed  through  Blackboard  to  all  students  enrolled  in  the  course.  Responses  to  the  questions  were 
anonymous  and  were  scored  on  a four-point  Likert  type  scale  from  “most  agree  with  the  description  (3)”  to 
“least  agree  with  the  description  (0)”.  The  respondents  were  also  given  the  opportunity  to  provide  additional 
comments. 

Instructor  survey.  The  instructors  were  administered  a paper-based  survey  that  was  similar  to  the 
student  survey.  Their  attitudes  towards  the  content  and  the  teaching  strategies  were  rated  on  the  same  four-point 
Likert  type  scale. 

Interview  protocol.  Interviews  with  the  instructors,  focus  group  students,  and  the  course  coordinator 
were  designed  to  collect  detailed  information  on  topics  covered  by  the  course,  teaching  strategies  used,  and 
student  learning  gain  at  the  end  of  the  semester.  The  questions  were  directly  aligned  to  the  questions  on  the 
survey.  All  interviewees  were  given  the  chance  to  address  any  issue  relevant  to  the  course. 

Test  scores.  Student  performance  on  the  online  quizzes,  online  midterm  exam,  and  hands-on  final  were 
analyzed  to  determine  the  usefulness  of  content  and  helpfulness  of  strategies.  The  online  tests  were  multiple  - 
choice  questions  that  assess  student  knowledge  about  computers  and  skills  of  using  a computer.  The  hands-on 
final  exam  tested  students  how  to  use  Microsoft  Word,  Excel,  and  PowerPoint.  Students  needed  to  complete 
editing,  analysis,  and  design  activities  as  requested.  The  four  10-point  quizzes  covered  the  primary  topics  of  the 
course,  namely:  Quiz  1 - File  Management,  Word,  Internet  and  World  Wide  Web,  Quiz  2 - IPSO,  Quiz  3 - 
Excel  and  Data  Analysis,  and  Quiz  4 - PowerPoint.  Topics  covered  during  the  first  half  of  the  semester  (Quiz  1 
and  2)  were  tested  in  the  online  midterm  exam.  The  maximum  score  of  the  online  midterm  and  the  Hands-on 
final  was  30  points. 


Results 

Survey  responses.  Based  on  a four-point  Likert -type  scale  scoring  from  3 (most  agree  with  the 
description)  to  0 (least  agree  with  the  description),  the  survey  responses  of  329  students  and  1 1 instructors  on 
the  usefulness  of  the  contents  are  shown  in  Table  1,  and  the  helpfulness  of  teaching  strategies  are  shown  in 
Table  2. 

In  terms  of  the  usefulness  of  the  contents  taught,  student  and  instructor  survey  responses  revealed  the 
topics  covered  in  this  course  are  generally  considered  very  useful  or  useful  (student  rating  M = 2.08,  instmctor 
rating  M = 2.44).  Among  the  seven  items  that  were  rated,  Microsoft  Word , PowerPoint,  Internet  and  World 
Wide  Web  were  the  top  three  highest  ratings  across  students  and  instructors.  Knowledge  on  IPSO  was  rated  the 
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least  useful  (student  rating  M = 1.53,  instructor  rating  M = 1.64).  It  is  worth  noting  that  there  was  a big 
diffeience  in  the  perceived  usefulness  on  File  Management  between  students  (M  = 1.74)  and  instructors  ( M = 
2.45). 

In  terms  of  teaching  strategies,  both  students  and  instructors  ranked  that  Hands-on  Projects , In-class 
Activities  to  Develop  Practical  Skills,  and  Handouts  for  Activities  and  Projects  were  the  three  most  helpful 
strategies.  Students  and  instructors  agreed  that  the  two  least  helpful  strategies  are  Readings  by  students  (student 
rating  M = .73,  instructor  rating  M = 1.09)  and  Online  Discussion  Forums  (student  M = 1.22,  instructor  rating  M 
= 0.82). 

The  responses  to  the  open-ended  question  at  the  end  of  the  survey  are  summarized  in  Table  3.  Twenty- 
nine  people  said  they  were  satisfied  with  the  course  and  no  change  needed  to  be  made.  There  were  also 
suggestions  in  terms  of  contents  taught  and  teaching  strategies  used.  Many  students  preferred  to  have  more 
hands-on  activities  in  class  (N  = 36)  and  get  more  individual  attention  from  the  instructor  (N  = 21).  Students 
also  commented  that  lectures  in  the  form  of  PowerPoint  presentations  were  too  long  (TV  = 19).  The  group  Online 
Discussion  Forum  was  considered  not  helpful  (N  = 12)  due  to  the  difficulty  of  cooperation  among  group 
members.  Students  also  considered  reading  textbook  as  not  helpful  ( N = 12),  and  the  quiz  questions  did  not  test 
the  skills  learned  but  how  much  details  remembered  (TV  =6). 

Interviews.  The  results  of  the  interviews  of  instructors  and  student  focus  groups  were  consistent  with 
those  of  the  survey  on  the  topics  covered  in  the  course.  In  addition,  both  students  and  instructors  emphasized 
that  more  practices  were  needed  for  File  Management.  Specific  step-by-step  instructions  were  needed  for 
complicated  subjects  such  as  Excel  and  Create  a Personal  Website. 

The  interview  with  the  course  coordinator  revealed  that  this  course  is  meant  to  provide  fundamental 
knowledge  and  skills  that  students  can  start  with.  Software  application  skills  (Microsoft  Word,  Excel, 
PowerPoint,  and  Create  a Personal  Website ) were  long-term  gains  for  the  students.  It  was  difficult  to  maintain 
consistency  of  teaching  across  21  sections  and  1 1 instructors.  PowerPoint  presentations  were  pre -designed  and 
distributed  to  instructors  to  maintain  the  consistency. 

Text  scores  The  mean  scores  and  percentages  of  achievement  of  all  tests  across  the  21  sections 
are  summarized  in  Table  4.  Among  the  online  tests,  students  performed  the  best  in  Quiz  2 on  IPSO  (M  = 8.14) 
and  Quiz  3 on  Excel  (M  = 8.09).  The  highest  achievement  was  their  hands-on  final  exam  (M  = 26.17),  which  is 
a 87. 23%  of  the  maximum  score. 

Discussion 

What  to  teach? 

According  to  the  survey  results,  the  contents  taught  in  the  course  achieved  a rating  of  2.26  across 
students  and  instructors,  indicating  that  the  topics  covered  by  the  Computer  Literacy  course  are  useful  or  very 
useful.  However,  there  appears  to  be  a big  difference  in  learner  interest  of  learning  knowledge  and  learning 
application  skills.  It  seems  that  application  skills  (M  = 2.44),  including  Word,  Excel,  PowerPoint,  and  Create  a 
Personal  Website,  were  rated  more  useful  than  concept  knowledge  (M  = 2.04)  that  consisted  of  File 
Management,  Internet  and  World  Wide  Web,  IPSO.  Does  this  mean  concept  knowledge  is  less  important  than 
application  skills? 

Interestingly,  the  concept  of  File  Management  was  rated  2.45  by  the  instructors,  0.7 1 point  higher  than 
student  rating  (M  = 1.74).  In  addition,  students  in  the  interview  suggested  more  practice  on  File  Management. 
This  might  be  because  many  students  were  not  able  to  realize  that  mastering  File  Management  is  essential  to 
working  with  their  documents  correctly  and  efficiently.  Poor  file  management  may  not  be  a problem  until  late  in 
the  semester  when  student  have  to  hyperlink  all  their  assignments  to  their  personal  websites.  If  this  is  the  case, 
how  can  the  concept  knowledge  be  taught  in  a more  interesting  way?  Can  the  knowledge  be  integrated  with  the 
teaching  of  application  skills? 

How  to  teach? 

The  survey  results  showed  that  all  of  the  three  hands-on  strategies  ( Hands-on  projects.  In-class 
activities,  Handouts  for  activities  and  projects ) were  helpful.  In  addition,  the  overall  rating  of  hands-on 
strategies  (M  = 2.53)  is  much  higher  than  the  overall  rating  of  other  strategies  (M  = 1 .48).  This  result  is 
consistent  with  student  and  instructor  interview.  It  indicates  thatskills  are  learned  through  hands-on  practice 
with  instructor  feedback.  In  order  to  give  students  more  attention  in  class,  the  class  size  should  be  small  enough 
for  the  instructor  to  provide  individual  assistance.  For  a class  that  has  18  to  24  students,  it  might  be  better  to 
downsize  the  class  if  there  is  only  one  instructor  available.  Alternatively,  there  can  be  a second  instructor  added 
to  assist  hands-on  practices. 

Both  the  survey  results  and  interview  revealed  that  the  current  Online  Discussion  Forum  is  not  a 
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preferred  teaching  strategy.  Some  student  reported  that  the  group  work  did  not  work  as  it  was  supposed  to.  This 
might  be  because  students  have  different  schedules  and  it  is  hard  to  work  as  a group  after  class.  To  improve 
collaboration  among  students,  it  is  advisable  to  provide  more  opportunities  in  class  to  work  together. 
Furthermore,  each  student  should  have  a clear  idea  of  his  or  her  responsibility  and  how  their  job  will  be  rated. 
Instructor  supervision  over  the  discussion  forum  needs  to  be  enhanced.  Guidance  and  feedback  provided  by  the 
instructor  will  also  motivate  students  to  do  a better  job. 

An  interesting  phenomenon  is  that  the  type  of  assessment  may  influence  student  performance.  As 
shown  in  Table  4,  student’s  achievement  in  the  online  tests  was  lower  than  the  final  hands-on  exam.  Students 
reported  that  the  online  quizzes  were  “too  detailed”  and  “did  not  test  the  skills  learned”  but  “how  much  is 
memorized”.  The  majority  of  online  test  questions  were  multiple -choice  format,  which  was  used  to  assess 
concept  knowledge  and  practical  application  skills.  Since  the  skills  would  be  tested  without  actually  using  the 
application  software,  the  multiple  -choice  format  seems  to  have  changed  the  nature  of  application  skills  from 
“hands-on”  to  memory -based.  Consequently,  student  performance  was  hampered  by  this  type  of  assessment. 
This  may  also  explain  why  Quiz  2 had  the  highest  mean  scores  among  all  the  online  tests  since  the  subject  of 
Input,  Processing,  Storage,  and  Output  was  mostly  factual  knowledge  as  opposed  to  application  skills  . Along 
the  same  line,  the  reason  why  Quiz  3 on  Excel  and  Data  Analysis  had  achieved  a high  average  might  be  because 
the  questions  were  not  just  theoretical  knowledge  but  it  also  tested  some  hands  on  excel  skills , and  students  had 
to  do  calculations  using  Excel  worksheet.  In  this  case,  students  could  actually  work  in  the  software  instead  of 
recalling  where  to  access  a command  and  under  which  menu.  It  appears  that  students  perform  better  when 
hands-on  skills  are  assessed  in  a hands-on  way. 

Recommendations 

The  Computer  Literacy  course  is  considered  as  a good  general  studies  course  that  provides  students  with  basic 
understanding  of  computer  and  application  skills  to  use  a computer.  Based  on  the  evaluation  results,  the 
following  recommendations  are  made: 

1.  Reinforce  the  understanding  of  concept  knowledge  in  teaching  application  skills. 

2.  More  in-class,  hands-on  activities  should  be  emphasized  in  teaching  facilitated  by  appropriate  student- 
instructor  ratio. 

3.  Online  discussion  forums  can  be  used  for  sharing  experience  of  using  application  software. 

4.  Use  hands-on  tests  to  assess  application  skills. 
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Table  1 Student  and  Instructor  Survey  Mean  Scores  on  the  Usefulness  of  Content 


Application  Skills 

Students 

Instructors 

Total 

Word 

2.53 

2.73 

2.63 

Excel 

2.13 

2.55 

2.34 

PowerPoint 

2.35 

2.82 

2.59 

Create  a Personal  Web  Site 

2.01 

2.36 

2.19 

Subtotal 

2.26 

2.62 

2.44 

Concept  Knowledge 

File  Management 

1.74 

2.45 

2.10 

Internet  and  World  Wide  Web 

2.30 

2.55 

2.43 

Input,  Processing,  Storage,  Output 

1.53 

1.64 

1.59 

Subtotal 

1.86 

2.21 

2.04 

Total 

2.08 

2.44 

2.26 

Table  2 Student  and  Instructor  Ratings  of  the  Helpfulness  of  Teaching  Strategies  in  the  Course 


Hands-on  Strategies 

Students 

Instructors 

Total 

Hands-on  projects 

2.52 

2.91 

2.72 

In-class  activities 

2.28 

2.64 

2.46 

Handouts  for  activities  and  projects 

2.27 

2.55 

2.41 

Subtotal 

2.36 

2.70 

2.53 

Other  Strategies 

PowerPoint  presentations  to  deliver  lecture 

2.12 

1.82 

1.97 

Online  multiple  -choice  tests 

1.84 

2.09 

1.97 

External  website  links 

1.69 

1.64 

1.67 

Co-operative  group  work 

1.34 

1.27 

1.31 

Online  discussion  forums 

1.22 

0.82 

1.02 

Reading  textbooks 

0.73 

1.09 

0.91 

Subtotal 

1.49 

1.46 

1.48 

Total 

1.78 

1.87 

1.83 
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Table  3 A Summary  of  the  Answers  to  the  Opened-ended  Question  in  Student  Survey 


Student  Comments 

Frequency 

1. 

Satisfied  with  the  content  taught;  the  course  was  well  organized;  no  changes  need 
to  be  made. 

29 

2. 

To  include  more  hands-on  practice/examples  or  step-by-step  instructions  in  class, 
especially  when  teaching  complicated  subjects. 

36 

3. 

Need  more  than  one  instructor  for  a class  size  of  18-24  people;  instructors  should 
give  more  individual  attention  to  students. 

21 

4. 

The  lectures  and  presentations  were  long  and  boring;  demonstrations  are  more 
interesting. 

19 

5. 

The  group  Online  Discussion  Forum  should  be  removed  or  changed;  the  group 
activity  designed  did  not  work. 

12 

6. 

Reading  from  the  textbook  had  little  value. 

12 

7. 

Split  the  weekly  three-hour  class  into  two  shorter  meetings. 

9 

8. 

The  quizzes  were  too  detailed;  the  quizzes  did  not  reflect  skills  learned  but  tested 
memorization. 

6 

9. 

To  include  more  collaborative  activities  to  make  the  class  more  interesting. 

6 

Table  4 Mean  Scores  of  Quizzes  and  Exams  across  All  Sections 


Quiz  1 

Quiz  2 

Midterm 

Quiz  3 

Quiz  4 

Hands-on  Final 

Word,  File 

Input, 

Including  Quiz 

Excel  and 

PowerPoint 

Word,  Excel, 

Management, 

Processing, 

1 and  Quiz  2 

Data  Analysis 

PowerPoint 

Internet  and 

Storage, 

WWW 

Output 

6.85 

8.14 

22.83 

8.10 

7.00 

25.57 

69% 

81% 

76% 

81% 

70% 

85.23% 
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An  Investigation  of  the  Relationships  Between 
Reading  Speed  and  Paper  Hue  Intensity,  Age  and  Gender 

Ireta  Ekstrom 

Virginia  Polytechnic  Institute  and  State  University 

Introduction 

Due  in  part  to  health  care  improvements  and  the  post  World  War  Two  Baby  Boom,  (Siegel,  1972),  the 
population  of  older  adults  is  growing  and  will  continue  to  grow.  By  2030  approximately  20  percent  of  the  U.S. 
population  will  be  over  age  65  (Verma,  1989).  By  attending  to  readability  (the  ease  of  reading  a printed  page) 
and  legibility  (the  speed  with  each  letter  or  word  can  be  recognized),  some  age-related  difficulties  may  be 
circumvented. 

Beginning  with  a 1931  study.  Tinker  and  Paterson  began  reporting  about  their  research  on  reading 
speed  and  other  reading  factors.  Additional  researchers,  especially  Stanton  and  Burtt  (1935)  added  to  that 
research.  Based  in  part  on  the  1931  study  Tinker  and  Paterson  (1940)  later  published  a little  research-based 
book  ( How  to  Make  Type  Readable.  A Manual  for  Typographers,  Printers  and  Advertisers,  1940)  which  defined 
color  combinations,  leading,  line  width  and  type  face  choices.  In  this  book,  Paterson  & Tinker  stated  that 
greater  contrast  between  paper  hue  and  ink  color  allows  improved  differentiation  of  letter  shapes  from  the 
background. 

A literature  review  revealed  several  factors  that  affect  gender-related  readability  and  legibility  for 
aging  adults.  The  literature  review  explored  five  relevant  areas:  (1)  aging  process  of  the  eye,  (2)  contrast 
sensitivity  with  aging,  (3)  research  utilizing  reading  speed  as  the  significant  criterion,  (4)  contrast  legibility 
studies  and  (5)  gender  and  readability  studies. 

Aging  Process  of  the  Eye 

Sight  allows  humans  to  learn  about  the  environment  through  reading,  movies,  television  and 
observation  of  others.  Vision  begins  when  light  enters  the  eye  via  the  transparent  lens,  is  changed  into  electrical 
signals  by  the  retina  which  sends  them  to  the  brain  where  they  are  interpreted.  Around  the  age  of  twenty,  each 
of  these  elements  beings  to  change  and  some  visual  functions  diminish.  Verner  and  Davison  (1982)  state  that 
defective  vision  increases  from  23  percent  at  age  twenty  to  95  percent  at  the  age  of  seventy. 

Accommodation  difficulty  changing  focus  from  near  to  distant  objects)  and  Presbyopia  (the  inability  to 
see  small  print  or  focus  on  nearby  images)  affect  all  aging  adults.  Around  age  65  almost  100  percent  of  adults 
cannot  focus  on  close  objects.  Other  causes  of  vision  changes  can  include  prescription  drugs,  environmental 
factors  and  various  diseases.  Health  problems  can  include  glaucoma,  cataracts,  miosis,  macular  degeneration 
and  diabetic  retinopathy  (Verma,  1989,  Lyle,  1974). 

Contrast  Sensitivity  with  Aging 

Perceiving  edges  and  sharp  outlines  allows  one  to  see  shapes  (Arden,  1978).  Caird  & Williamson 
(1986)  found  that  decreasing  contrast  sensitivity  can  be  a major  cause  of  visual  difficulty  especially  in  dim  light 
or  under  glaring  lights.  Several  studies  have  used  contrast  sensitivity  as  the  preferred  method  of  measuring 
vision.  Owsley,  Sekuler  and  Siemsen  (1983)  found  that  contrast  sensitivity  function  is  a good  predictor  of  a 
subject's  visual  performance.  They  surmise  that  it  can  probably  be  attributed  to  the  amount  of  light  reaching  the 
retina. 

Reading  Speed  as  a Significant  Criterion 

Legge,  Rubin,  Pelli,  et  al.  (1988),  Brown  (1981),  Cooper  (1985)  utilized  reading  speed  as  a major 
criterion  in  studies.  Their  results  found  that  reduced  contrast  or  low  light  levels  can  create  problems  for  most 
low  vision  subjects.  Reading  speed  is  also  easier  and  more  straightforward  to  use  than  legibility  as  an  objective 
measure  (Legge,  Rubin,  Pelli  & Schleske,  1985a;  Legge,  Rubin  & Luebker,  1987;  Legge,  Rubin,  Pelli,  et  al., 
1988). 


290 


Contrast  Legibility  Studies 

Paterson  and  Tinker  stated  in  1940  that  printers  should  have  the  maximum  contrast  between  the  ink 
and  background.  The  best  arrangement  is  a dark  hue  for  the  letters  and  a light  hue  for  the  background  (for 
example,  dark  blue  or  black  for  the  ink  and  white  or  cream  for  the  paper). 

Gender  and  Readability  Studies 

Three  international  studies  found  that  women  were  significantly  more  likely  to  have  vision  problems 
with  age.  The  Attebo,  Mitchell  and  Smith  (1996)  study  (the  Blue  Mountains  Eye  Study  from  Australia)  found 
that  seventy  nine  percent  of  the  persons  with  severe  visual  impairment  over  49  years  of  age  were  female. 

Taylor,  Livingston,  Stanislavsky  and  McCarty  (1997)  reported  that  Australian  urban  women  older  than  40  years 
of  age  had  significantly  higher  rates  of  blindness.  Age  adjusted  rates  of  blindness  were  .066%  in  men  and  .17% 
in  women.  The  West,  Munoz,  Rubin,  Schein,  Bandeen- Roche,  Zeger,  German  and  Fried  (1997)  study  from 
Baltimore,  Maryland  (the  Salisbury  Eye  Evaluation  Project),  confirmed  higher  rates  of  vision  loss  in  women 
which  affected  daily  life  and  could  contribute  to  more  women  in  care  settings. 

These  studies  form  the  foundation  for  this  study  that  measured  contrast  sensitivity  as  people  age.  While 
many  people  have  researched  legibility  and  readability,  no  studies  have  defined  gender-related  contrast 
sensitivity  as  it  correlated  to  paper  of  various  hues  and  intensities  for  aging  adults. 

This  study  extended  Paterson  and  Tinker’s  (1940)  and  Stanton  and  Burtt’s  (1935)  research  to  include 
older  adults  and  also  to  determine  if  there  are  gender  differences  in  the  ability  to  read  on  various  hue  intensities. 
It  utilized  updated  statistical  methods,  a wider  age  range,  gender  identification,  the  use  of  a modern  reading  test 
and  modern  paper  choices  and  test  replication. 

The  specific  research  questions  addressed  in  this  study  were: 

1.  To  what  extent  does  paper  hue  intensity  affect  the  reading  speed  of  persons  of  varying  ages? 

2.  To  what  extent  do  paper  hue  intensity  and  gender  affect  the  reading  speed  of  persons  of  varying  ages? 

3.  What  are  the  relationships  between  reading  speed  and  paper  hue  intensity,  age  and  gender? 

Methods 

Research  Design 

This  study  utilized  a repeated  measure  single  subject  design  and  sampled  adults  aged  twenty  and  over. 
This  research  added  middle  aged  and  older  adults  to  Stanton  and  Burtt’s  1935  study.  Keppel  and  Zedeck  (1989) 
define  the  repeated  measures  single  subject  design  as  "an  experiment  in  which  [all]  subjects  are  each  tested 
under  all  treatment  conditions"  (p.  267). 

Advantages  of  a repeated  measure  single  subject  design  included: 

1.  A smaller  error  term,  since  by  using  one  subject  the  natural  differences  between  subjects  did  not  need  to  be 
part  of  the  factor. 

2.  Individual  differences  were  controlled  since  each  subject  served  as  his  or  her  own  control,  thus  increasing 
homogeneity. 

3.  A significantly  reduced  need  for  subjects  (Keppel  & Zedeck,  1989). 

To  lessen  the  disadvantages  of  this  design  (sensitization  effects,  practice  effects  and  carry-over  effects) 
measures  were  taken  such  as  counterbalancing  the  treatment  order,  creating  awareness  of  the  purpose  and 
design  of  the  study  and  providing  specific  directions  to  participants. 

The  independent  variable  studied  was  the  hue  intensity  of  paper.  Dependent  variables  studied  were 
gender,  age  and  reading  speed.  Controlled  variables  included  room  environment,  type  size  and  the  ability  to 
read  and  vision  problems. 

One  hundred  fifty  subjects  were  determined  to  be  statistically  necessary  for  this  study.  Equal  numbers 
of  male  and  female  subjects  (75  each)  were  evenly  divided  into  five  age  groups  with  an  age  range  was  76  years 
(20  years  to  96  years).  Age  groupings  were  20-32,  33-45,  46-58,  59-71  and  72-96.  The  mean  age  was  51.83 
years  old  and  the  median  age  was  50  years.  The  standard  deviation  was  17.31. 

Ethnic  backgrounds  of  the  sample  included  Caucasian  (88.7%),  African  American  (7.3%),  Native 
American  and  Pacific  Islander  (1.3%  each),  Arabic  (.7%)  and  other  nationalities  (.7%).  Education  levels  of  the 
subjects  included  less  than  high  school  or  finished  high  school  (13.3  percent),  some  college  (32.7  percent), 
bachelor’s  degree  (28.0  percent)  and  graduate  degree  (26  percent). 

The  convenience  sample  was  drawn  from  a church  in  a northern  Detroit  suburb  (n=l  19)  and 
educational  settings  in  a hospital  and  two  universities  (n=31). 

The  three  colors  used  in  the  research  were  black-and-white  photographed  with  an  artist’s  gray  scale. 
The  formula  “(Letters  - Uackground)  (Iietters  + Uackground)”  (Legge,  Rubin,  & Luebker,  1987)  was  utilized  for  a 
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contrast  number.  The  ink  color  equaled  black  and  the  paper  hue  equaled  the  number  from  the  formula.  The 
paper  hues  with  the  greatest  range  (blues,  reds  and  greens)  were  chosen  for  this  research.  Their  ranges  were 
.086,  .134  and  .134.  The  Metropolitan  Achievement  Test  (MAT6)  (1985)  was  used  to  measure  reading  speed. 

Research  data  consisted  of  demographic  data,  reading  speed  scores  and  paper  hue  intensity.  The 
statistical  analysis  used  for  questions  one  and  two  was  a one  factor  repeated  measures  analysis  of  variance.  It 
tested  for  age  and  gender-related  reading  speed  differences  on  the  three  intensities  of  colored  papers.  The 
statistical  analyses  for  question  three  were  correlations  plus  stepwise  regressions.  They  were  conducted  to 
explain  differences  in  the  dependent  variables. 


Results 

Question  One 

To  answer  question  one,  “To  what  extent  does  paper  hue  intensity  affect  the  reading  speed  of  persons 
of  varying  ages?”,  paper  hue  intensity  scores,  MAT6  reading  speed  scores  and  the  five  age  groups  were  needed. 
A general  factorial  ANOVA  was  used  as  a test  of  significance.  Table  1 identifies  the  mean  number  of  lines  read 
on  all  three  paper  intensities 

Table  1 

Means  of  Lines  Read  by  Age  Groups  on  Light,  Medium  and  Dark  Intensities 


Means 

Ages  in 
Group 

Light  Intensity 

Medium  Intensity 

Dark  Intensity 

20-32 

31.77 

33.53 

36.16 

33-45 

40.67 

36.50 

37.93 

46-58 

35.60 

34.77 

35.37 

59-71 

34.67 

33.10 

35.57 

More  than  72 

29.50 

31.80 

29.80 

Scores  on  light  intensities  indicated  that  the  three  light  intensities  are  statistically  diverse  and  that  there 
are  also  differences  in  the  reading  speed  scores  of  the  five  age  groups.  An  interaction  was  also  found  between 
reading  speed  scores  on  the  three  light  paper  intensities  when  combined  with  age.  Table  2 presents  the  results 
of  the  analysis  of  variance  among  the  light  paper  intensities,  age  groups  and  reading  speed. 


Table  2 


The  Relationships  Among  Age  Groups  and  Reading  Speed 
Scores  on  Light  Intensities  of  Paper 


Source  of  Variation 

Significance  of  F 

Light  Paper  Intensities 

.000* 

Age  Groups 

.023* 

Light  Intensity  by  Age  Groups 

.000* 

*p<  .05 

No  statistically  significant  relationship  among  age,  medium  intensities  of  paper  and  reading  speed  was 
found.  Table  3 presents  the  results  of  the  analysis  of  variance  among  the  medium  paper  intensities,  age  groups 
and  reading  speed. 

Table  3 

The  Relationships  Among  Age  Groups  and  Reading  Speed  Scores 


on  Medium  Intensities  of  Paper 

Source  of  Variation 

Sig.  ofF 

Medium  Paper  Intensities 

.283 

Age  Groups 

.060 

Medium  Intensities  by  Age  Groups 

.065 

*p<  .05 
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Scores  on  dark  intensities  indicated  statistical  diversity.  An  interaction  was  also  found  between  reading 
speed  scores  on  the  three  dark  paper  intensities  when  combined  with  age.  Table  4 presents  the  results  of  the 
analysis  of  variance  among  the  dark  paper  intensities,  age  groups  and  reading  speed. 

Table  4 


The  Relationships  Among  Age  Groups  and  Reading  Speed  Scores 
on  Dark  Intensities  of  Paper 


Source  of  Variation 

Sig.  ofF 

Dark  Paper  Intensity 

.008* 

Age  Groups 

.364 

Dark  Intensities  by  Age  Groups 

.044* 

*p<  .05 

The  first  research  question  addressed  the  relationships  among  paper  hue  intensity,  reading  speed  and 
age.  Light  and  dark  intensities  varied  significantly  among  age  groups.  In  this  research,  reading  speeds  on  both 
light  and  dark  intensities  of  paper  hue  became  slower  (fewer  lines  read)  as  the  adults  aged.  Medium  intensity 
had  no  significant  variance  indicating  that  reading  speed  did  not  vary  significantly  with  these  paper  hue 
intensities. 

Question  Two 

The  second  research  question  considered  the  question  of  paper  hue  intensity  and  gender  affecting  the 
reading  speed  of  persons  of  varying  ages.  A general  factorial  ANOVA  was  employed  to  compare  the  reading 
speed  scores  on  each  of  the  three  paper  intensities  with  the  five  age  groups.  No  statistical  interaction  was  found 
among  the  three  hues,  age  groups  and  gender  (Tables  5). 

Table  5 


Means  of  Lines  Read  by  Gender  on  Light , Medium  and  Dark  Intensities 


Means 

N=150 

Gender 

Light  Intensity 

Medium  Intensity 

Dark  Intensity 

N 

Male 

33.63 

35.41 

35.19 

75 

Female 

35.25 

32.47 

34.75 

75 

Participants'  reading  speed  scores  on  light  intensities  of  paper  hues  indicated  that  a statistical 
interaction  was  found  between  the  three  light  hues  and  age  groups.  The  ANOVA  results  in  Table  6 illustrate  the 
data  regarding  light  paper  intensities. 

Table  6 


The  Relationships  Among  Reading  Level  Scores,  Light  Intensities  of  Paper,  Age  Groups  and  Gender 


Source  of  V ariation 

SS 

DF 

MS 

F 

Sig.  of  F 

Within  + Residual 

8297.60 

120 

69.15 

Hue  Name 

419.68 

2 

209.84 

3.03 

.052* 

Age  Groups 

2151.56 

4 

537.89 

7.78 

.000* 

Gender 

99.23 

1 

99.23 

1.44 

.233 

Light  Paper  Intensity  by  Age  Group 

1248.52 

8 

156.07 

2.26 

.028* 

Light  Paper  Intensity  by  Gender 

117.49 

2 

58.75 

.85 

.430 

Age  Groups  by  Gender 

149.51 

4 

37.38 

.54 

.706 

Light  Paper  Intensity  by  Age  Groups  by  Gender 

411.37 

8 

51.42 

.74 

.653 

(Model) 

4597.36 

29 

158.53 

2.29 

.001 

(Total) 

12894.96 

149 

86.54 

*P  < .05 
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Scores  on  the  medium  intensities  indicated  a statistically  significant  interaction  with  gender.  Table  7 
highlights  the  relationship  between  gender  and  medium  intensity  paper  hues  and  presents  the  complete  analysis 
of  variance  results. 


Table  7 


The  Relationships  Among  Reading  Level  Scores,  Medium  Intensities  of  Paper,  Age  Groups  and  Gender 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig.  of  F 

Within  + Residual 

9309.60 

135 

77.68 

Hue  Name 

228.76 

2 

114.38 

1.47 

.233 

Age  Groups 

380.63 

4 

95.16 

1.23 

.303 

Gender 

325.61 

1 

325.61 

4.20 

.043* 

Hue  Name  by  Age  Groups 

1153.57 

8 

144.20 

1.86 

.073 

Hue  Name  by  Gender 

55.61 

2 

27.81 

.36 

.700 

Age  Groups  by  Gender 

167.03 

4 

41.76 

.54 

.708 

Hue  Name  by  Age  Group  by  Gender 

209.65 

8 

26.21 

.34 

.950 

(Model) 

2520.86 

29 

86.93 

1.12 

.326 

(Total) 

11830.46 

149 

79.40 

*£  < .05 


Scores  on  the  dark  intensities  indicated  a statistically  significant  interaction  with  age  and  gender.  Table 
8 highlights  the  relationship  between  gender  and  age  and  dark  intensity  paper  hues  and  presents  the  complete 
analysis  of  variance  results. 


Table  8 

The  Relationships  Among  Reading  Level  Scores,  Dark  Intensities  of  Paper,  Age  Groups  and  Gender 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig.  of  F 

Within  + Residual 

8909.20 

135 

74.24 

Hue  Name 

154.49 

2 

77.25 

1.04 

.356 

Age  Groups 

1123.67 

4 

280.92 

3.78 

.006* 

Gender 

7.26 

1 

7.26 

.10 

.755 

Hue  Name  by  Age  Groups 

687.17 

8 

85.90 

1.16 

.331 

Hue  Name  by  Gender 

504.76 

2 

252.38 

3.40 

.037* 

Age  Groups  by  Gender 

171.11 

4 

42.78 

.58 

.680 

Hue  Name  by  Age  Groups  by  Gender 

909.17 

8 

113.65 

1.53 

.154 

(Model) 

3557.63 

29 

122.68 

1.65 

.032 

(Total) 

12466.83 

149 

83.67 

*P  < .05 


Reading  level  scores,  paper  intensity,  age  group  and  gender  were  examined  in  question  two.  An 
analysis  of  variance  for  light  paper  intensities  indicated  an  interaction  among  age  groups.  For  medium 
intensities,  gender  was  significant  and  age  and  gender  were  significant  for  dark  paper  intensities. 

Question  Three 

To  answer  question  three,  three  stepwise  multiple  regressions  correlated  each  paper  hue  intensity  with 
participants'  age  and  gender.  Negative  results  indicated  that  older  adults  read  fewer  lines  in  the  time  allotted. 
Therefore,  they  experienced  increased  difficulty  when  reading  on  both  light  and  dark  paper  intensities  (Tables  9 
and  10). 
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Table  9 


The  Relationships  Between  Three  Light  Paper  Hue  Intensities  and 
Reading  Level,  Age  & Gender 


B T 


Age 

(Constant) 

r .18 

R2  .032 

F 4.95 

Sig  F .028 


-.086 

38.98 


-2.23 

17.93 


SigT 

.028 

.000 


Table  10 


The  Relationships  Between  Three  Dark  Paper  Hue  Intensities,  Age  & 
Gender 


B 

T 

SigT 

Age 

-.11 

-3.01 

.003 

r 

R2 

F 

Sig  F 

(Constant) 

.24 

.06 

9.08 

.003 

40.93 

19.40 

.000 

Gender  also  was  negatively  correlated  with  medium  paper  intensities.  Table  11  summarizes  medium 
paper  hue  intensities  and  their  relationships  between  reading  speed  and  gender.  The  negative  results  indicated 
that  women  read  fewer  lines  than  men  in  the  time  allotted.  Therefore,  women  experienced  increased  difficulty 
when  reading  on  all  three  hues’  medium  intensities. 

Table  1 1 


The  Relationships  Between  Three  Medium  Paper  Hue  Intensities,  Age 
& Gender 


B 

T 

SigT 

Gender 

-2.947 

-2.05 

.043 

(Constant) 

38.36 

16.85 

.000 

r .17 

R2  .03 

F 4.19.043 

Sig  F 

T-tests  were  conducted  separately  with  each  light,  medium  and  dark  paper  hue  (green,  red  and  blue).  A 
Scheffe'  test  (at  a significance  level  of  .05)  found  no  two  groups  were  significantly  different  at  the  .05  level. 

This  indicated  that  the  three  light,  medium  and  dark  paper  intensity  means  were  not  significantly  different;  they 
were  homogeneous  subsets. 

Correlations  between  participants'  reading  speed  on  the  green  hues  indicated  that  participant's  age  and 
gender  were  correlated  with  the  dark  green  paper  intensity.  This  negative  correlation  indicated  that  older  women 
experienced  difficulty  when  reading  on  the  dark  green  paper  hue.  No  correlations  were  found  with  the  blue 
paper  hue. 

When  stepwise  regressions  were  conducted,  participants'  reading  speed  scores  were  correlated  with  the 
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three  paper  intensities.  Participants'  ages  were  significant  with  light  and  dark  paper  intensities.  Medium  paper 
intensity  was  correlated  with  gender. 

Table  12 

Summarized  Findings  of  Research  Questions 

Question  Source  of  Data  Significant  Relationships 


1.  To  what  extent  does 
paper  hue  intensity 
affect  the  reading 
speed  of  persons  of 
varying  ages? 


MAT6  Reading  Test 
Scores 

Participants'  ages 


2.  To  what  extent  do 
paper  intensity  and 
gender  affect  the 
reading  speed  of 
persons  of  varying 
ages? 

3.  What  are  the 
relationships 
between  reading 
speed  and  paper  hue 
intensity,  age  and 
gender? 


MAT6  Reading  Test 
Scores 

Participants'  gender 


MAT6  Reading  Test 
Scores 

Participants'  ages 
Participants'  gender 


Relationships  among  reading  speed  scores,  age  and  paper 
intensity  were  significant  for: 

> light  intensities  of  paper  with  age  groups. 

> dark  intensities  of  paper  with  age  groups. 

Relationships  among  paper  intensity,  gender  and  age  were 
significant  for 

> light  intensities  of  paper  and  age  groups. 

> mediumintensities  of  paper  and  gender. 

> dark  intensities  of  paper  and  gender  with  age 
groups. 

Stepwise  regressions  discovered  significant  relationships 
between  participants'  age  and  reading  speed  on  the  light  and 
dark  red  paper  hues. 

Participants'  reading  speed  on  the  dark  green  paper  hue 
demonstrated  a significant  relationship  with  age  and  gender. 
Blue  indicated  no  significant  relationships. 

The  light  and  dark  paper  shades  were  significantly  related 
to  participants'  age. 

Participants'  reading  speed  on  medium  shades  was 
significant  with  gender. 


Discussion 

It  is  becoming  increasingly  important  to  create  materials  that  are  easy  to  read  for  the  aging  population 
as  this  group  will  soon  will  encompass  25  percent  of  the  population.  Lifelong  learning  as  well  as  recreational, 
health,  financial  and  legal  reading  are  important  for  this  group  of  aging  adults.  Unless  care  is  taken  to  ensure 
that  the  printed  pages  are  readable  and  legible,  they  are  useless.  More  attention  is  being  paid  to  older  adults  in 
designing  many  products  (for  example,  automobiles  and  homes).  Implications  of  this  research  may  influence  a 
photocopying  business,  typesetter  or  printing  company's  ability  to  aid  customers  as  well  as  health  care, 
education  and  recreational  reading. 

While  it  appears  sensible  to  produce  materials  that  all  persons  can  utilize,  it  is  not  a commonly 
accepted  practice.  These  questions  are  universally  needed  for  a complete  picture  of  printed  materials  and  aging. 
Two  additional  questions  might  be: 

1.  Does  type  size  affect  the  reading  speed  of  adults  of  varying  ages  as  they  read  on  an  assortment  of  paper 
hue  intensities  ? 

2.  What  are  the  effects  of  line  length  on  reading  speed  as  people  age? 
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Introduction 

As  the  whole  society  is  experiencing  a notable  shift  from  the  industrial  age  to  the  information  age,  an 
urgent  need  for  a mindset  change  in  education  has  been  frequently  discussed  during  the  past  decades.  This 
paper  will  approach  the  mindset  change  through  three  interconnected  sections:  the  first  section  reviews  the 
conceptualization  of  mindset  and  then  gives  our  definition  of  mindset  concentrating  on  understanding  its  unique 
significance  to  the  educational  system.  The  second  section  presents,  compares,  and  contrasts  the  key  markers  of 
the  information-age  mindset  and  the  industrial-age  mindset.  The  third  section  displays  an  instrument  designed 
and  developed  by  the  authors  that  can  be  used  to  measure  the  status  of  individual  and  group  mindset. 

Definitions  and  Significance  of  Mindset 


What  is  a mindset? 

Since  this  paper  is  a part  of  a transforming  education  system  project,  we  need  to  define  our  “mindsets” 
concept  in  order  to  explore  the  interaction  and  interrelationships  between  peoples’  mindsets  and  systemic 
change  in  education.  Webster’s  dictionary  (n.d.)  defines  mindset  in  two  ways:  the  first  is  “a  mental  attitude  or 
inclination”  and  the  second  is  “a  fixed  state  of  mind”.  Both  definitions  explain  the  mindset  as  something  that 
occurs  in  a person’s  head;  however,  the  mindset  also  has  the  power  to  control  a person’s  attitudes,  and 
potentially  influences  a person’s  behavior. 

In  order  to  have  a deeper  understanding  of  the  mindset  and  its  importance,  our  exploration  of  the 
mindset  definition  goes  beyond  a literal  explanation.  A few  related  terms  are  examined,  including  paradigm, 
belief,  and  worldview.  These  concepts  provide  insight  into  the  meaning  of  mindset. 

Kuhn  defined  and  popularized  the  concept  of  paradigm  in  his  remarkable  The  Structure  of  Scientific 
Revolutions,  in  which  he  argues  that  a paradigm  consists  of  “rules  that  guide  particular  normal  tradition”  (pl6). 
In  many  areas  including  education,  a paradigm  is  a model  or  an  exemplar  that  forms  the  foundation  that 
prepares  people  for  professional  practice.  In  most  social  structures,  paradigms  are  well-defined  and  rigid.  And 
when  people  accept  a paradigm,  it  becomes  a firmly  fixed  belief  in  their  mind.  However,  paradigm  shift  does 
take  place  when  an  anomaly  undermines  the  basic  beliefs  underlying  the  basic  practice.  The  paradigm  shift 
doesn’t  refer  to  a piecemeal  change  or  a minor  modification  of  the  current  practice,  instead  it  is  a revolution,  a 
transformation,  or  a kind  of  metamorphosis.  In  a word,  it  is  a fundamental  change  from  one  way  of  thinking  to 
another. 

Mindset  sometimes  can  be  simply  exp  ressed  as  “I  (we)  believe.”  Belief  is  the  major  component  of  the 
mindset,  because  people  set  up  their  expectations  and  goals  based  on  what  they  believe  the  nature  of  situations 
should  be.  Their  activities  are  guided  by  their  beliefs  and  they  are  inclined  to  act  to  implement  the  paradigm  in 
the  life.  For  example,  learners’,  educators’,  and  administrators’  beliefs  about  what  schools  should  achieve 
determines  the  face  of  school  systems,  such  as  the  design  of  schools’  key  characteristics  and  their  major 
functions. 

A worldview  describes  “a  consistent  and  integral  sense  of  existence  and  provides  a framework  for 
generating,  sustaining  and  applying  knowledge.”  (WorldlQ,  n.d.).  Mindset  understood  through  the  worldview 
perspective  focuses  on  the  interaction  between  the  mindset  owners  and  the  global/local  environments.  Mindset 
here  is  “a  habitual  or  characteristic  mental  attitude  that  determines  how  people  will  interpret  and  respond  to 
situations”  (HyperDictionary,  2003).  People  originate  their  worldview  from  the  unique  experience  they  have 
perceived,  by  their  own  or  through  heritage,  and  then  forge  their  opinions  and  engage  in  activities  based  on  their 
worldview.  For  example,  Shantz  and  Rideout  (2003)  pointed  out  that  the  industrial-age  worldview  might 
interpret  learning  as  “possession  of  knowledge  and  ability  contained  in  the  overt  curriculum”  (p  203),  and  thus 
stress  the  standardization  and  centralization  in  schools. 

Our  definition  denotes  a comprehensive  set  of  the  mindset  components  that  are  the  basis  of  people’s 
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cognitive  process  and  their  operational  guidance.  Therefore,  we  define  mindsets  as  the  basic  assumption, 
beliefs,  core  values,  goals  and  expectations  shared  by  a group  of  people  who  are  committed  to  a specific  field, 
and  what  they  will  use  as  rules  to  guide  their  attitudes  and  practice  in  the  field. 

Why  are  mindsets  important? 

Kuhn  said  that  a shift  in  paradigm  occurs  as  a response  to  crisis.  In  most  cases,  a crisis  begins  from  the 
awareness  of  an  anomaly  in  a changing  environment,  such  as  the  emergence  of  new  objects.  If  the  existing 
paradigms  fail  to  explain  the  emerging  phenomena  or  solve  the  new  puzzles,  a search  for  creative  or  adaptive 
responses  will  be  demanded.  During  the  past  decades,  there  have  been  massive  changes  around  the  world, 
stimulated  by  the  advancement  of  information  technology.  These  changes  present  a number  of  key  markers  that 
distinguish  the  emerging  new  age,  the  information  age,  from  the  industrial  age  (Reigeluth,  2001).  As  the 
industrial-age  mindset  still  pervades  the  ways  we  see  the  world  and  we  do  things,  these  changes  force  us  to 
question  the  industrial-age  mindset  and  its  capability  to  solve  the  information-age  puzzles.  For  example,  the 
business  world  has  found  that  the  traditional-centralized  organization  is  obsolete  for  providing  customized 
services  demanded  by  the  information-age  customers,  so  shifts  have  take  place  in  business  culture  from  working 
around  departments  to  working  around  process  that  provide  value  to  the  customers  (Hammer  & Champy,  1993). 

In  exploring  the  complexities  of  human  history,  Toffler  has  seen  three  great  waves:  the  first 
agricultural  revolution  that  began  to  move  people  from  hunting  to  peasant  societies  about  10,000  years  ago;  the 
second  industrial  revolution  which  took  the  shape  in  the  18th  century  when  people  began  to  leave  the  peasant 
culture  of  farming  to  come  to  work  in  city  factories;  and  now  the  whole  society  is  undergone  a third  wave  that 
is  transforming  the  industrial  era  to  a knowledge-based  era,  triggered  by  the  rise  of  the  analog  and  then  digital 
technologies.  The  current  transition  from  brute  force  to  brain  force  accompanies  a painful  dislocation  of  all  the 
aspects  of  our  lives.  Institutions  that  were  designed  to  work  in  a factory -based  society  are  gasping  for  their  last 
breath. 

The  history  of  educational  reform  has  proved  that  when  a society  is  undergoing  fundamental  changes, 
education  needs  to  change  fundamentally,  too.  A good  example  is  the  systemic  change  that  took  place  in 
education  when  we  moved  from  the  one  roomschoolhouse  to  industrialized  mass  education  during  the 
transition  from  agriculture  age  to  the  industrial  age.  The  Lancastrian  model  of  education  made  it  possible  to 
provide  a large  amount  of  skilled  labor  for  the  factory -centered  society.  However,  when  the  third  wave  came 
along,  the  information  age,  the  prevailing  mindset  in  education,  as  well  as  in  many  other  social  facets,  was 
unable  to  generate  effective  solutions  to  the  problems  of  the  information  age.  Because  of  this,  another  mindset 
change  is  urgently  needed. 

The  predominant  industrial  paradigm  of  education  became  entrenched  in  the  early  20th  century. 

During  the  past  hundred  years,  learning  has  been  defined  as  the  acquisition  of  knowledge  or  ability  in  a 
separately  enclosed  learning  environment  by  a group  of  learners.  It  requires  learners’  adherence  and  obedience 
to  the  covert  curriculum  and  the  design  of  schooling  is  a typical  conformity  system:  diverse  learners  learn  same 
content  at  the  same  time  and  with  the  same  rate.  (Shantz  & Rideout,  2003) 

The  report  A Nation  at  Risk  (1983)  discloses  a number  of  risks  the  industrial-age  education  system  has 
generated  in  the  information  era:  from  high  dropout  rate  to  low  academic  achievement,  from  enlarging  the 
functional  illiterate  population  to  a declined  proportion  of  exceptional  performance  in  standard  tests.  With  a 
continuously  improved  awareness  of  the  crisis  of  education,  many  crucial  questions  have  been  raised  about  the 
influence  of  information-age  characteristics  on  education.  The  implications  of  the  new  trends  have  been 
examined  in  educational  technology,  instructional  theory,  and  instructional  systems  technology  and  a call  for 
the  emergence  of  an  information-age  mindset  could  be  heard  more  and  more,  including  the  transformation  from 
a closed  system  to  an  open  one,  from  a bureaucratic  approach  to  a team-based  one,  and  from  a student -screening 
focus  to  a learner-enabling  one  (Banathy,  1992;  Cornell,  1999;  Deuchar,  2004;  O’Neill,  2003/2004;  Reigeluth, 
1999;  Solomon,  2000). 

In  the  21st  century,  which  will  be  dominated  by  information-age  trends,  it  is  important  for  educators  to 
recognize  the  industrial-age  mindset  that  limits  our  thinking  and  empower  the  education  system  through  a 
redesign  or  transformation  to  the  information-age  mindset. 

Information-Age  Mindset  vs.  Industrial -Age  Mindset 

In  the  section  one  we  have  briefly  discussed  the  movement  to  the  information-age  mindset  from 
industrial-era  mindset.  However,  many  may  still  wish  clarification  about  the  distinction  between  these  two 
mindsets.  This  section  will  focus  on  what  are  the  key  markers  of  these  two  mindsets  and  contrast  their  features 
in  order  to  generate  a deep  understanding  of  the  two  mindsets’  impact  on  education  system  design. 
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As  transforming  from  the  industrial  age  to  the  information  age,  tremendous  changes  have  taken  place 
in  the  society’s  structure  and  the  way  people  behave  themselves  in  their  lives.  Passi,  Michelet  and  Passi  (2003) 
compare  and  contrast  the  industrial  age  and  the  information  age  from  a broad  scope  of  interests,  including 
economy,  family,  social  activities,  technology,  education,  among  other  things.  Table  1 displays  the  major 
characteristics  that  the  authors  have  used  to  distinguish  the  two  eras. 


Wealth 

Dimensions  of  Life 
Habitat  and  Social  Unit 

Social  Classes 
Warfare 

Economy 
Energy  Source 

Production 

Family 

Education 

Position  Determined  by 
Management 


Industrial  Era 

Industry /Tangible  Assets 

Earth  + Universe 

Mega-city  Static  (tied  to  factory) 

Owners,  workers 
Mechanical  technology,  mass 
destruction 
National/International 
Non-renewable:  fossil,  fuels,  gas, 
oil,  coal 

Mass  -produced  and  distributed 
(cheap)  for  others 
Small,  nuclear,  mobile 

Mass -education,  same  for  all  (3 
Rs) 

Wealth 
By  rationality 


Information  Era 
Information/Knowledge 
Earth  + Universe  + Cyberspace 
Global  community  Mobile  (link 
through  Web) 

By  level  of  knowledge  and  ideas 
(Dis)  Information  technology, 
targeted  destruction  for  control 
Global,  vet  local 
Renewable:  solar,  nuclear,  laser 

Mass -customized  (cheap)for 
others 

Individual,  mono-parental, 
mobile 

Individualized  and  differentiated 
for  all 

By  responsibility  (?) 


Table  1:  Key  Markers  of  Industrial  Age  and  Information  Age.  Adapted  from  Passi,  et.  al.  (2003). 


The  changes  occurring  in  the  environment  have  a great  impact  on  the  whole  human  culture.  Key 
markers  of  both  the  industrial  age  and  the  information  age,  such  as  core  values,  principles,  philosophies,  and 
organizational  management,  have  been  described  and  analyzed  as  to  distinguish  these  two  eras  (Berge,  2003; 
Gordon,  Morgan,  & Ponticell  1994;  Hanna,  2003;  Reigeluth,  2003).  Table  2 depicts  these  key  markers  and  how 
they  represent  the  two  fundamentally  different  ages. 


Key  markers  (global  items)  Industrial  Age 


Information  Age 


Philosophy 

Relationship 


Value 

Decision  making 
Communication 

Process 


Quality  control 


Standardization 
Mass  production,  etc. 
Adversarial  relationships 
CEO  or  boss  (teacher)  as  “king” 
Compliance 

Bureaucratic  organization 
Conformity  (compliance) 
Autocratic  decision  making 
One-way  communications 
Compartmentalization 
Parts  oriented  (division  of  labor) 

Planned  obsolescence 
Centralized  control 


Customization 
Customized  production 
Cooperative  relationships 
Customer  as  “King” 

Initiative 

Team-based  organization 
Diversity 

Shared  decision  making 

Networking 

Holism 

Process  oriented  (integration  of 
tasks) 

Total  quality 

Autonomy  with  accountability 


Table  2:  Key  markers  that  distinguish  industrial  age  and  information  age  organization.  Sources  from: 
Reigeluth  (1999,  2001,  n.d.).  Key  marker  added. 
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Education,  as  any  other  social  aspect,  is  unavoidably  influenced  by  the  changes  triggered  by  the 
information  age.  Table  3 presents  a comparison  between  the  industrial  age  education  and  the  information  age 
education,  particularly  as  these  differences  correspond  to  philosophy,  leadership,  teaching,  assessment,  and 
relationship. 


Key  markers  (education-focused 
items) 

Industrial  Age 

Information  Age 

Philosophy 

Focus  on  teaching 

Focus  on  learning 

Teaching  (caring  about)  a subject 

Teaching  (caring  about)  Caring 
about  the  whole  student 

All  students  learn  the  same  thing 

Cultivate  special  talents  of  each 
students 

Leadership 

Command  & control  leadership 

Participatory /empowering 
leadership 

Teaching 

Students  progress  at  the  same 

Each  students  progresses  upon 

pace  (students  all  learn  the  same 

mastery  (Students  learn  what  they 

thing  at  the  same  time) 

ready  to  learn  when  they  are 
ready) 

Focus  on  presenting  all  the 

Focus  on  learning  all  the 

materials 

materials 

Grade  levels 

Continuous  progress 

Covering  the  content 

Outcomes -based  learning 

Group-based  content  delivery 

Personal  learning  plans 

Classrooms 

Learning  centers 

Teacher’ s role 

Teacher  as  sage  on  the  stage, 

Teacher  as  guide  on  the  side, 

Teacher  as  dispenser  of 
knowledge  of  learning 

Teacher  as  coach  or  facilitator 

Purpose 

Only  the  good  students  succeed 

All  students  succeed 

Assessment 

Non-authentic  assessment 

Performance-based  assessment 

Norm-based  testing  (compared 

Criterion-based  testing  (compares 

students  to  each  other) 

students  to  a standard) 

Norm-referenced  testing 

Individualized  testing 

Relationship 

Adversarial  learning 

Cooperative  learning 

Separate  from  community 

Community  involvement 

Parents  as  occasional  spectators 

Parents  as  partners 

Content 

Memorization  of  meaningless 

Thinking,  proble  m-solving  and 

facts  skills 

meaning  making 

Isolated  reading,  writing  skills 

Communication  skills 

Key  markers  related  to  education-focused  items.  Sources:  Reigeluth  (2004  ). 


As  shown  in  the  above  tables,  we  see  a total  departure  of  information-age  mindset  from  the  industrial- 
age  key  markers.  The  recognition  of  the  information-age  mindset  not  only  forces  us  as  educators  to  think  about 
their  implication  on  the  education,  but  also  enable  us  to  redesign  today’s  schools  in  order  to  serve  learners 
diverse  potentials  and  multiple  intelligences. 

An  Instrument  of  Measuring  Minds 

Why  the  measurement  of  mindset  is  important? 

As  the  mindset  change  in  education  is  gaining  more  and  more  proponents,  we  have  little  appreciation 


301 


and  few  criteria  for  measuring  the  status  of  mindset.  Too  little  of  the  education  literature  provides  clear  criteria 
for  assessing  the  positions  of  mindset  in  a practical  continuum.  So  it  is  the  focus  of  this  section  to  advance  clear 
criteria  that  others  can  use  in  evaluating  people’s  mindset  and  to  what  extent  and  in  what  manner  it  is  going  to 
change. 

The  context  for  the  instrument  application 

Mindset  change  has  been  discussed  tremendously  in  various  contexts,  from  sociology  to  pure 
technology,  and  we  have  found  that  there  is  a lack  of  consistence  regarding  the  definition  and  application  of  the 
mindset  concept  in  all  these  contexts.  The  measurement  of  mindset  also  has  different  criteria  under  various 
circumstances.  In  order  to  enhance  the  practicability,  the  instrument  we  created  would  like  to  limit  the 
boundaries  only  in  the  education  field,  with  a concentration  on  systemic  change. 

A tool  for  measurement 

Mindset  indicators  Based  on  this  definition,  people’s  cognition,  affection  and  behaviors  are  engaged  in  the 
notion  of  mindset.  In  other  words,  if  somebody  has  the  information-age  mindset  (or  the  industrial  age  mindset) 
about  education,  he/she  may  have  explicit  and/or  implicit  knowledge  about  the  information-age  (or  the 
industrial-age  mindset)  through  experiences  and/or  learning.  He/she  also  may  have  positive  attitudes  and 
affective  evaluation  about  his/her  mindsets  and  its  relevant  events  or  situations  whereas  he/she  may  have 
relatively  negative  attitudes  towards  irrelevant  situations.  In  addition,  naively  speaking,  his/her  consistent 
behaviors  with  his/her  mindset  can  be  also  expected.  In  three  domains  (cognitive  domain,  affective  domain,  and 
behavioral  domain)  all  factors  such  as  applications  of  cognition  to  situations,  affective  reactions  such  as 
favorability  or  affective  evaluation,  and  intention  of  behaviors  can  be  indicators  of  people’s  mindsets. 

Cognition  and  its  application  to  various  situations.  Mindsets  are  unlike  human  traits;  therefore,  they  are  not 
innate  but  learned.  Current  mindsets  that  people  possess  are  products  of  learning  from  their  experiences,  and  for 
mindset  evolution,  formal  or  informal  learning  processes  need  to  be  engaged.  During  learning  processes  for 
mindset  evolution,  people  should  acquire  notions  of  new  mindsets,  and  by  evolving  mindsets  they  can  apply 
new  mindsets  to  relevant  situations  naturally.  In  addition,  mindsets  are  relatively  consistent  frameworks  to  see, 
accept,  and  understand  situations,  and  events.  Therefore,  people’s  knowledge  and  especially,  its  application  to 
various  situations  can  be  an  appropriate  indicator  to  show  people’s  mindset 

Based  on  this  point,  mindset  about  education  may  be  measured  using: 

• Artificial  scenarios  which  contain  conflicts  between  information-age  mindset  and  industrial-age 
mindset  about  education,  and  people’s  open-ended  responses. 

• Small  group  discussion  with  real  cases  or  scenarios  about  education  evaluated  by  discussion  peers 
and/or  third  party  evaluators  who  are  knowledgeable  about  mindset  change  in  education. 

Affective  reactions  When  measuring  cognition  and  its  application  to  various  situations,  it  may  be  helpful  to 
measure  affective  reactions  and  evaluation  about  the  situations  since  mindset  is  not  only  about  cognitive 
processes  but  also  about  people’s  affective  attitudes.  In  other  words,  people  may  tend  to  be  more  favorable  to 
situations  which  are  more  relevant  with  their  mindsets.  People  with  the  information-age  mindset  will  rate  that 
situations  or  decisions  which  support  the  information-age  mindset  are  more  ‘favorable’,  ‘appropriate’, 
‘beneficial’  and/or  ‘correct’  than  those  related  to  industrial-age  mindset,  and  vice  versa. 

Based  on  this,  as  one  of  the  indicators  of  the  mindset  about  education  affective  reactions  can  be 
measured  using; 

• People’s  self  report  concerning  possible  decisions  about  scenarios  (for  cognition  application 
above)  using  5 point  Likert  scale  for  example;  inappropriate  - appropriate,  unfavorable -favorable, 
incorrect-correct,  not-preferable  -preferable,  etc. 

• (if  possible)  after  the  small  group  discussion  about  real  cases  or  scenarios  about  education,  peers 
and/or  third  party  evaluators  assess  people’s  affective  attitudes  towards  situations  and  cases 

Intention  of  behaviors  One  of  the  ultimate  goals  of  mindset  change  is  people’s  behavior  change.  However,  it  is 
hard  to  measure  actual  behaviors  based  on  evolved  mindset  because  many  studies  about  attitudes,  and  theories 
of  reasoned  behaviors  show  that  it  takes  time  to  reflect  people’s  mindsets  including  cognitions  and  attitudes  to 
their  behaviors,  and  more  complex  processes  and  variables  involve  in  behavior  changes.  Therefore,  in  the 
current  situation,  instead  of  actual  behavior  change,  the  intention  of  behaviors  can  be  preferably  measurable. 
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To  reflect  ideas  of  behavior  intention  in  the  scenarios  above,  questions  need  to  be  asked  using  the  first 
party  viewpoint  instead  of  the  third  party  view  point  such  as  “if  you  were  in  the  situation,  how  would  you 
behave?” 

Self-report  vs.  measurement  by  others  The  self  report  method  to  measure  mindset  is  one  of  the  possibilities. 
Through  the  self-report  method  such  as  responding  to  given  scenarios  people’s  understanding  level  about 
mindset  and  its  components  can  be  clearly  disclosed.  In  spite  of  this  strength  it  is  dangerous  to  use  self-report 
as  the  only  measurement  because  social  preference  can  contaminate  measuring  the  mindset.  Currently  the 
targeted  subjects  to  measure  mindsets  are  leadership  team  members,  and  they  might  respond  with  what  they 
have  learned  in  the  leadership  team  of  Systemic  Change  in  Decatur  School  District  and  with  preferable  answers 
of  the  team  rather  than  individual  beliefs  intentionally  or  unintentionally.  The  issue  of  intentional  deception  to 
respond  can  be  reduced  by  emphasis  of  self-disclosure  and  the  purpose  of  measurement  which  is  not  for 
evaluation  but  for  diagnosis  and  help.  However,  the  issue  of  unintentional  deception  in  responding  --  people 
reflect  knowledge  they  learn  in  the  team  instead  of  their  beliefs--  can  still  be  problematic  in  self-report  method. 

To  complement  the  weakness  of  self  reporting,  it  would  be  preferable  to  use  the  measurement 
conducted  by  others.  In  regular  meetings  and  small  group  discussions  peers  and  knowledgeable  third  party 
observers  can  assess  people’s  mindsets,  and  the  data  from  others  can  be  triangulated  with  self-reports.  This 
triangulation  can  provide  higher  reliability  of  data. 

In  summary,  in  reference  of  measurement  tools  firstly  scenarios  with  open-ended  questions,  Likert 
scales  for  emotional  reaction,  and  questions  about  intention  of  behavior  will  be  used  in  self  report.  Secondly, 
small  group  discussions  and  rubrics  for  peer  evaluation  and  third  party  evaluation  will  be  used  to  complement 
the  self-report.  The  rubrics  will  contain  standards  for  people’s  knowledge  levels,  application  of  knowledge,  and 
emotional  reactions. 

Type  of  Indicators 

measurement  Applications  of  cognition  Affective  reactions  Intention  of  behaviors 

Self-report  Open-ended  scenarios  Affective  adjectives  rating  Open-ended  scenarios 

using  Likert  scale  about  the 
scenarios 

Measurement  Small  group  discussions  Attitudes  toward  situations 

by  others  and  peer- third  party-  and  topics  in  discussion 

evaluation 

Table  4.  Measurement  Methodology 


Conclusion 

As  massive  changes  are  forcing  the  whole  society  moving  toward  the  information  age, 
education  will  prepare  students  for  a world  that  doesn’t  exist  anymore  if  it  is  still  dominated  by  the  industrial 
age  mindset.  However,  before  we  can  know  how  the  mindset  will  change,  we  must  be  aware  of  the  current 
status  of  the  mindset.  In  order  to  facilitate  the  mindset  shift  in  education,  this  paper  creates  a three-indicators 
instrument  that  provide  the  practical  criteria  for  measuring  mindset,  and  therefore,  help  educators  to  gain  more 
insights  about  the  relevance  between  mindset  and  systemic  change. 
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Abstract 

This  paper  reports  on  a qualitative  study  of  student  Self  Regulated  Learning  behaviors  in  a video  - 
centric,  multimedia  course.  Of  interest  was  how  student  viewing  habits,  generally,  and  their  manipulation  of 
playback  controls,  specifically,  supported  their  learning.  Data  revealed  that  the  use  of  playback  controls  to 
regulate  learning  was  largely,  but  not  always,  purposeful,  and  that  numerous  variables  including  fatigue, 
intetface  design  and  course  deadlines  influenced  self-regulated  learning  behavior.  The  research  suggests  that 
self-regulated  learning  can  be  supported  in  digital  video-based  instruction  by  increasing  control  ajfordances  in 
the  intetface.  In  addition,  explicit  strategies/instruction  should  be  given  to  learners  on  how  such  affordances 
might  help  them  regulate  their  learning. 

Statement  of  the  Problem 

I have  long  held  that  instruction  and  learning  can  be  both  inhibited  and  promoted  with  the  use  of 
technology.  Some  of  this  is  due  to  features  of  the  technology  itself,  with  other  variables  being  dependant  on  the 
learner.  That  is,  people’s  perceptions  and  attitudes  towards  technology  can  be  equally  inhibiting  and  facilitative 
to  learning.  Self-Regulated  Learning  (SRL)  generally  stipulates  that  students  “personally  activate  and  sustain 
behaviors,  cognitions,  and  affects,  which  were  systematically  oriented  toward  the  attainment  of  learning  goals” 
(Schunk,  2004  p.355).  SRL  also  describes  how  learners  purposefully  control  the  interactions  between 
themselves,  their  learning,  and  their  environment  (Bandura,  1986;  Zimmerman,  1989). 

The  nature  of  video-centric  instructional  environments  pose  some  particular  challenges  to  designing 
for  SRL.  The  video  medium’s  “linear-ness”  makes  skimming  and  purposeful  navigation  through  video  (and 
audio)  materials  a challenge.  Of  particular  interest  in  this  study  was  the  control  affordance  of  Variable  Speed 
Playback  (VSP)  or  the  ability  to  control  the  playback  speed  of  digitized  audio  and  video.  Learners  can  now 
listen  to  audio/video-centric  multimedia  instruction  at  their  desired  speed — speeds  they  deem  conducive  to  their 
learning,  regardless  of  how  quickly  or  slowly  a lecturer  originally  spoke,  or  whether  developers  ever  intended 
the  materials  to  be  accelerated.  This  study  served  to  explore  how  VSP  and  other  course  features  facilitated  or 
inhibited  SRL  behaviors.  Our  understanding  of  SRL  specific  to  various  computing  technologies  is  very  limited 
(Winne  & Stockley,  1998)  and  scarcely  addresses  these  types  of  questions. 

It  is  perhaps  obvious  that  the  locus  of  control  for  SRL  behavior  remains  within  the  learner.  This  in 
turn,  points  to  the  perhaps  less -obvious  deduction  that  the  environment  is  really  the  only  element  over  which 
instructional  designers  have  significant  direct  control  to  help  support  student  learning.  Schunk  (2004)  states 
that  “the  potential  for  self-regulation  varies  depending  on  control  choices  available  to  learners”  (p.  122).  Thus, 
if  students  are  to  exercise  self -regulation,  efforts  must  be  made  to  design  video-centric  environments  with  high 
levels  of  learner-control  affordances,  thereby  providing  learners  with  control  over  how,  when,  what  and  where 
they  engage  in  learning  activities. 


Background 

Variable  Speed  Playback  (VSP) 

As  the  use  of  VSP  emerged  as  such  a significant  self  regulating  tool  in  this  study  and  in  this  context  of 
video-centric  instruction,  I will  devote  some  time  here  addressing  the  issue  of  VSP  directly.  Today’s  VSP 
technology  allows  one  to  speed  up  and  slow  down  audio  and  video  presentations  without  pitch  distortions  or  the 
high  speed  “chipmunk”  sound  associated  with  accelerating  audio  cassettes  and  CDs — or  reaching  way  back — 
our  old  vinyl  records.  It  has  recently  experienced  a resurgence  in  both  availability  and  popularity  and  is 
included  now  by  default  in  Microsoft’s  Windows  Media  Player. 

In  an  earlier  survey  study  on  VSP  functionality  (Galbraith  & Spencer,  2001),  students  reported 
regularly  accelerating  through  instructional  presentations  up  to  2.5  times  (2.5x)  the  normal  playback  speed  by 
the  end  of  the  semester.  Through  difficult  to  provide  the  reader  with  a sense  of  what  2.x  speed  means,  it 
suffices  to  say  that  it  is  nearly  unintelligibly  fast  to  the  untrained  ear.  At  the  same  time,  a few  students  choose 
to  use  no,  or  very  little,  acceleration.  Their  self-reported  motivations  varied  widely.  Importantly,  VSP  differs 
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from  fixed-rate  acceleration  in  that  users  can  dynamically  adjust  video  presentation  speed  to  individually 
comfortable  rates.  It  was  hoped  to  learn  how,  if  at  all,  students  might  be  varying  playback  speed — a regulating 
behavior— in  response  to  their  perceived  learning.  Studies  have  shown  (Harrigan,  1995)  that  giving  the  user  or 
learner  control  over  the  rate  of  acceleration  is  generally  better  than  “locking”  acceleration  in  at  a given  rate. 

In  my  years  as  a professional  non-linear  video  editor  I learned  the  value  of  being  able  to  swiftly 
navigate  and  view— “speed  read”— vast  amounts  of  media  at  high  speeds  while  still  being  able  to  comprehend  the 
content.  In  my  more  recent  years  as  an  instructional  media  designer/developer  I built  VSP  functionality  into  the 
accounting  course  allowing  students  to  dynamically  control  the  playback  rate  or  speed  at  which  the  instructional 
presentations  (audio,  video,  graphics  and  animation)  played.  It  is  important  to  note  that  prior  to  introducing 
VSP  to  the  multimedia  course,  student  surveys  indicated  quite  positive  attitudes  toward  the  course.  Their 
single  biggest  frustration,  however,  was  that  the  professor  spoke  too  slowly  and  repeated  himself  too  often.  I 
recall  some  students  asking— pleading-for  a way  to  “speed  him  up”.  As  courseware  developers,  I and  my  team 
were  personally  all-too-aware  of  the  student’s  sentiment.  We  had  just  spent  months  listening  to  these  materials, 
videotaping,  digitizing,  editing,  programming,  testing,  revising,  retesting  the  course  materials  and  had  listened 
to  more  than  our  share  of  the  content!  A solution  to  accelerate  the  hypermedia  lectures  would  benefit  the 
developers  as  much  as  the  students. 

We  were  capable  and  had  considered  accelerating  the  course  videos  by  a fixed  or  pre  -determined 
percentage,  but  how  much  should  we  speed  things  up?  Was  the  effort  worth  a global  10%  speed  up?  Would 
that  hurt  some  learners?  It  was  determined  that  accelerating  the  audio  by  a pre -determined  percentage  was  not 
in  the  best  interest  of  students,  and  that  the  solution  would  have  to  give  learners  control  over  the  playback 
speed.  A solution  was  identified  (2xAV  plugin  from  Enounce  Inc.)  that  allowed  learners  to  dynamically  adjust 
the  playback  speed  to  suit  their  preference  and  it  was  immediately  integrated  midway  through  the  school 
semester.  Course  feedback  and  positive  ratings  skyrocketed  that  semester — we  were  on  to  something!  We  had 
struck  something  valuable,  but  were  not  sure  what;  and  did  not  fully  understand  its  properties  nor  its  effects. 

As  the  designer,  I was  very  pleased  that  since  introducing  VSP  functionality,  student  frustration  levels 
had  subsided,  and  I hoped  now  that  motivation  and  comprehension  might  also  increase,  and  that  students  would 
use  acceleration  responsibly.  I feared,  however,  that  the  positive  response  might  have  been  simply  due  to  the 
fact  that  they  could  “whip”  through  the  material  faster  than  before.  I was  left  with  questions  as  to  when,  where, 
how  and  why  students  might  use  VSP  technologies  to  support  and  regulate  their  learning.  Some  survey 
responses  clearly  reflected  use  of  VSP  as  a regulatory  strategy  to  help  maintain  attentiveness  and 
comprehension.  Other  student  responses  showed  signs  of  regulatory  practice,  but  not  in  support  of  learning — 
rather  time  management.  As  feared,  some  students  appeared  to  be  using  VSP  to  make  up  for  procrastination 
using  the  tool  to  simply  skim  through  materials  before  a quiz  deadline  or  to  avoid  missing  a bus. 

All  the  literature  I had  come  across  on  accelerated  audio  focused  primarily  on  listener  comprehension 
of  accelerated  speech,  or  the  effectiveness  of  various  speech  compression  algorithms,  but  not  the  usage  patterns- 
-and  certainly  not  couched  in  the  context  of  self-regulated  learning.  I found  one  usability  study  published  by 
Microsoft  that  evaluated  such  technology  in  a modified  version  of  an  early  media  player  (Omoigui,  He , Gupta, 
Grudin,  & Sanocki,  1999).  Like  other  studies,  it  did  not  assess  learner  intent  and  motivation— why  users  were 
motivated  to  adjust  speeds?  but  it  did  record  and  report  on  how  their  viewers  interacted  with  5 video  samples 
that  varied  in  content  and  duration,  over  the  period  of  a couple  days  of  viewing. 
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Figure  1 shows  how  the  researchers  tracked  average  speeds  of  users  for  every  10%  segment  of  the 
video.  Of  their  observations,  they  noted,  “...we  clearly  see  that  the  subjects  are  watching  them  faster  as  they 
get  deeper  into  the  video.  There  is  some  slowdown  right  at  the  end,  an  area  that  corresponds  to  the  concluding 
remarks”  (Omoigui  et  ah,  1999,  p.5).  I was  generally  imp  ressed  with  their  study  and  with  the  detailed 
observation  data  they  were  able  to  collect,  but  it  was  the  last  line  about  an  implied,  but  not  corroborated, 
connection  between  viewing  habits  and  concluding  or  summary  remarks  that  really  stood  out  to  me.  I knew  that 
I wanted  similar  kinds  of  data  to  the  Microsoft  study.  It  would  help  us  understand  first  how — and  only 
thereafter,  why  users  were  interacting  with  course  features  the  way  they  did  (specifically  VSP  functionality). 
Moreover,  as  an  instructional  designer  and  educational  researcher,  I desired  to  understand  how  purposeful  such 
regulatory  behavior  was. 


Methods 

The  qualitative  research  approach  looked  to  be  an  appropriate  method  to  explore  the  nature  of  student 
self-regulated  learning  behaviors  using  course  features.  It  was  a way  to  help  the  implicit  emerge  and  become 
explicit;  a way  to  generate  hypotheses  from  social  research  data  that  is  systematically  obtained  and  analyzed 
(Glaser  & Strauss,  1967).  The  study  looked  at  a small  group  of  learners  in  an  entry  level  accounting  course  at  a 
large  western  university.  The  course  was  a computer-based,  video-centric,  multimedia  course,  comprising  of 
video  lectures  recorded  expressly  for  this  format,  supported  by  supplemental  media  and  simple  animations  that 
were  synchronized  with  the  lectures.  The  study  was  designed  to  better  understand  SRL  behavior  at  a micro 
level  (within  individual  lessons)  including  how  that  behavior  both  influenced  and  was  influenced  by  more 
macro  level  SRL  behavior  related  to  study  habits  and  time  management.  I made  one  hour  observations  of 
students  studying,  followed  by  another  40-60  minutes  of  interview.  Numerous  other  spontaneous,  non-formal 
observations  were  made  of  other  accounting  students  in  the  computer  labs  whom  I did  not  subsequently 
interview. 

The  software  and  course  design  provided  learners  with  a good  amount  of  control  and  autonomy  with 
the  bounds  of  a tradition  four-month  semester.  For  example,  at  the  macro  level,  the  course  was  largely  self- 
paced,  with  weekly  quizzes  scheduled  to  help  students  stay  on  target  (a  level  of  structure  deemed  necessary 
from  earlier  course  trials).  Students  were  provided  with  course  CDs  allowing  them  to  work  most  any  place  they 
had  access  to  a computer.  The  class  met  only  six  times  during  the  semester,  with  one  class  section  dispensing 
with  face-to-face  meetings  altogether.  At  the  micro  level,  in  addition  to  traditional  playback  controls  (play, 
pause  stop,  rewind)  the  courseware  interface  also  provided  a detailed  hyperlinked  index/menu  of  the  lesson 
segments,  independent  controls  for  supplemental  media  (slides),  and  what  proved  to  be  highly  relevant  to  SRL, 
was  a variable  speed  controller  allowing  learners  to  dynamically  adjust  the  playback  rate  of  the  course 
materials. 

Researcher  Identity 

To  be  sure,  I am  no  “disinterested”  or  objective  observer.  My  interest  in  this  topic  is  personal.  It 
springs  from  many  of  my  own  experiences  and  observations,  some  of  which  I have  already  described.  I was  the 
chief  designer  of  the  software  used  to  deliver  the  instruction  some  four  years  prior  to  this  study.  This  study  was, 
however,  not  a formative  evaluation  of  the  courseware,  nor  a summative  effectiveness  study.  Instead,  it  hoped 
to  focus  on  learner  self-monitoring  abilities  and  their  regulating  practices  and  strategies.  Study  participants 
were  likely  not  affected  by  my  having  designed  their  course  software — because  this  was  not  disclosed  to 
them — but  this  is  not  to  say  that  the  research  as  a whole  was  unaffected.  As  the  researcher,  I found  it  difficult  at 
times  to  stay  out  of  the  usability-testing  mode,  and  remain  focused  on  my  primary  research  question.  My  focus, 
unlike  in  previous  studies,  and  on  other  projects,  was  no  longer  looking  for  ways  to  improve  a particular  product 
or  software,  but  rather  to  understand  the  learners  and  how  they  employed  features  of  this  course  to  support  their 
learning. 

Access  & Participants 

I arranged  access  to  the  students  through  the  instructor,  with  whom  I had  worked  closely  in  the  past  to 
develop  instructional  tools.  By  way  of  reminder,  unlike  many  large  200  level  college  courses,  this  course  did 
not  meet  regularly  in  a classroom.  Instead  students  studied  course  materials  on  their  own.  The  mode  of 
instruction  arguably  requires  a good  deal  more  self-regulation  than  traditional  face  to  face  classes. 

I chose  not  to  disclose  to  students  that  I had  been  the  designer  of  the  software  they  were  using.  This 
was  a conscious  decision  on  my  part  because  I did  not  want  that  information  to  unduly  influence  their 
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conversations  with  me.  I believe  that  most  students  simply  perceived  me  as  I had  been  introduced  to  them — an 
alumnus  conducting  research  at  my  alma  mater. 

Eleven  participants  were  selected  from  among  a host  of  volunteers.  Volunteers  responded  to  either  an 
announcement  made  in  class  by  their  instructor,  or  to  the  same  announcement  posted  on  their  course  website. 

In  volunteering,  they  were  to  indicate  the  following  through  email.  1)  when  they  generally  studied-what  days, 
what  time,  what  environment?  and  2)  How  often  they  adjusted  speed  controls  (hardly  ever,  sometimes,  a lot).  I 
quickly  received  over  100  volunteers,  and  knew  I could  only  ever  deal  with  up  to  15  participants.  Generally,  I 
attempted  to  get  a variety  of  cases  that  would  likely  generate,  to  the  fullest  extent,  as  many  diverse  properties  of 
the  categories  as  possible  (Glaser  & Strauss,  1967  p.49). 

When  I had  “down  time”  between  scheduled  observation/interviews,  I ended  up  “trolling”  the 
computer  labs  on  campus  for  prospective  participants.  As  the  course  enrolls  over  1,200  students  each  semester, 
finding  students  in  this  manner  presented  no  significant  challenge.  At  any  given  time  of  day,  the  large  labs  I 
visited  had  at  least  two-six  people  working  on  the  accounting  course.  As  an  interesting  side  note,  on  March  31st 
I wrote  in  my  journal: 

Today  (3/31/04)  one  Acc  lab  TA  I talked  to  mentioned  that  I should  contact  evening  TAs 
as  it  is  they  who  deal  with  students  who  may  have  procrastinated  trying  to  prepare  for  and 
take  quizzes  that  are  due  by  midnight  (every  Tuesday  and  Thursday  night).  Interestingly, 
it  is  those  very  students  who  might  not  have  time  to  talk  to  me,  and  they  would  be  an 
interesting  group  to  include.  I did  not  seek  out  people  who  were  “early  birds”,  but  seem 
to  have  found  people  that  were  staying  well  ahead  of  the  game  with  the  exception  of  only 
one  I think. 

As  I noted  in  my  journal,  there  was  a group  of  alleged  “procrastinators”  with  whom  I did  not  speak. 
They  might  have  been  struggling  with  self-regulatory  practices  and  whose  perspective  would  certainly  have 
added  to  this  story,  but  were  unfortunately  not  included.  Despite  this,  ecological  validity  was  still  important  for 
me.  That  is,  it  was  important  for  me  to  observe  participants  in  the  environments  that  they  commonly  used  to 
study.  Observations  and  interviews  took  place  in  student  apartments,  a home,  an  office  and  in  various  campus 
computer  labs.  In  my  journal,  I recorded  some  of  my  thoughts  on  three  of  my  participants  whose  names  I have 
changed. 

Dave  was  an  interesting  find.  I overheard  that  he  was  an  accounting  major,  and  was 
surprised  because  of  the  amount  of  questions  he  asked  of  the  acc  TAs  in  the  lab.  He  also 
seemed  to  be  struggling  with  simple  concepts.  It  was  later  when  I approached  him  that  I 
learned  he  was  an  accounting  major.  I also  observed  that  he  was  viewing  materials  at  1.2 
and  1.5x  [comparatively  slow].  Again,  surprising  since  he  was  an  accounting  major  and 
should  be  “getting  this  stuff”.  While  speed  use  is  no  race,  I was  surprised  to  see  a[n 
accounting]  major  proceeding  so  slowly  and  deliberately.  It  is  good  to  see  a tool  that  is 
so  flexible,  and  can  accommodate  many  different  types  of  learners  and  their  self- 
regulation practices. 

Chris  was  also  of  particular  interest  to  me.  Chris  is  a handsome  trendy-looking  young 
man.  I had  run  into  his  father  on  campus,  an  old  casual  acquaintance  of  mine.  His  father 
told  me  of  how  Chris  was  taking  the  class  for  the  second  time  and  that  Chris  was 
diagnosed  as  having  ADD— but  was  currently  not  taking  any  medication.  Since  failing  is 
so  hard  to  do  in  this  class,  [in  my  opinion]  and  since  his  ADD  would  be  an  interesting 
case  in  relation  to  self-regulation,  I was  interested  in  his  story  and  arranged  to  observe 
and  interview  him  at  his  home  the  next  day. 

Troy  was  anxious  to  talk  to  me.  In  response  to  my  recruitment  email,  he  said  he  had 
opinions  on  the  course,  and  had  taken  many  distance  education  and  technology  courses.  I 
thought  his  perspective  might  be  interesting. . . for  one,  simply  because  he  seemed  so 
anxious  to  talk  to  me!  I expected  to  get  an  earful!,  and  not  necessarily  about  the  topic  of 
my  research.  Troy  was  a 40+-  year  old  professional  considering  an  executive  MBA 
program.  He  seemed  eager  to  continue  his  education,  perhaps  to  secure  a more  stable 
life.  Because  I had  followed  a career  path  similar  to  his,  and  sought  some  stability  for  my 
family,  perhaps  I am  projecting  my  rationale  onto  him.  We’ll  see. 

These  were  all  interesting  people  to  say  the  least,  but  I don’t  get  the  sense  that  these  are  extreme  cases. 
Everyone  in  the  class— all  1,200  of  them- -probably  has  equally  interesting  backgrounds  and  stories  that  bring 
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them  to  this  course.  I felt  extremely  privileged  to  be  allowed  into  my  participant’s  minds  and  their  study  time. 


Findings 

In  this  section,  I will  begin  to  describe  what  appears  to  be  emerging  from  the  data— a preliminary 
description  of  some  major  themes  found  in  the  interviews.  As  not  all  the  data  has  been  thoroughly  reviewed  at 
this  time  of,  findings  are  not  reflective  of  the  more  thorough  data  analysis  processes  included  in  open,  axial,  and 
selective  coding  recommended  for  such  a study  by  Strauss  & Corbin  (1998). 

Students  in  general  appeared  to  actively  self- monitor — and  quite  deliberate  about  how  they  used 
course  playback  controls . Controlling  the  speed  seemed  to  play  an  important  early  role  in  their  regulatory 
behavior  of  all  the  participants.  But,  once  comfortable  speeds  were  identified  fewer  speed  adjustments  were 
made  within  lessons  and  other  control  affordances  such  as  repetition,  became  more  dominant.  They  did  not 
adjust  very  often, (not  dynamically)  and  chose  instead  to  repeat  sections  rather  than  slow  down.  I wondered  out 
loud  with  one  participant  if  perhaps  an  acceleration  foot  pedal  might  be  a nice  device  to  have  for  speed 
control — sewing  machine  like.  Would  such  a device  encourage  more  dynamic  and  frequent  employment  of 
VSP  as  a self-regulated  learning?  The  worst  case  scenario  for  using  VSP  controls  meant  that  a participant  had 
to  set  aside  their  notes  from  off  their  lap,  lean  physically  forward,  clear  a space  on  the  desk  for  the  notes,  put 
their  pen  down,  grab  the  mouse,  navigate  to  the  VSP  controls  and  then  make  an  adjustment.  It  was  not  a 
necessarily  easy  and  natural  task,  yet  most  participants  situated  themselves  such  that  regulating  playback  speed 
in  relation  to  their  comprehension,  took  far  less  effort. 

Another  recurring  self- monitoring  and  regulating  theme  was  regarding  attentiveness  and  concentration. 
“Speeding  up  helps  me  stay  focused  and  keeps  my  attention  better  than  normal  speed”  said  Susie.  “It  saves 
time”  said  another  participant. 

“I  mean  the  quicker  I can  get  through  the  lesson  the  better.  But  I also  want  to  understand 
it,  you  know.  At  first  when  I started  doing  it,  I started  at  normal  speed  but  that  just  drove 
me  nuts  because  it  just  seemed  so  slow.  So  then  I put  it  on  double  speed  and  that  worked 
good  for  a while  and  then  it  just  seemed  like  it  was  too  slow  too,  so  I sped  up  to  about  2.2 
and  that  seemed  to  work  out  good.  Also,  like,  it  forces  me  to  focus  and  to  concentrate  be 
cause  it’s  going  so  fast  that  if  I don’t— like  if  I doze  off  or  something  I'll  miss  so  much. 

Whereas  if  it's  just  on  normal  speed,  it’s  kind  of  monotonous,  it’s  easy  to  not  focus  your 
thoughts,  so  I think  it  does  kind  of  help  you  to  focus  when  its  going  faster. 

Repeating  (replaying)  lesson  sections  became  a more  dominant  way  of  regulating  comprehension  than 
dynamic  and  frequent  speed  adjustments.  Repeating  still  all  occurred  at  higher  than  normal  (lx)  speeds.  Jack 
mused  that  he  expected  his  mind  wandered  less  at  higher  speeds  and  that  it  actually  reduced  the  number  of 
times  he’d  have  to  rewind  and  repeat  phrases  or  thoughts.  This  sentiment  seemed  universally  held  although  a 
couple  did  relate  equal  mind  wandering  when  viewing  too  fast.  Both  points  are  borne  out  in  the  literature.  (See 
Harrigan  1995,  2000;  Gutenko,  1995;  and  King  & Behnke,  1989  for  a discussion  on  these  issues.)  In  either 
case,  when  participants  felt  they  missed  particular  content,  they  chose  more  often  to  repeat  a few  lines  rather 
than  adjust  speed. 

This  of  course,  does  not  shed  light  on  the  speed  adjustments  that  did  occur.  Two  of  my  three  female 
participants,  Laura  and  Trisha,  adjusted  speed  a couple  different  times  during  the  lesson  for  related  reasons. 
Laura  started  out  her  lesson  in  the  accounting  lab  by  setting  her  speed  at  2. Ox  (2  times  normal  speed  -or  double 
speed).  After  about  a minute,  she  released  the  mouse  and  sat  back  in  her  seat,  listening  with  her  course  packet 
notes  open  in  front  of  her.  She  jotted  down  notes  now  and  then,  seemingly  following  along  with  the  lectures. 
After  about  six  minutes  she  leans  forward  and  increased  her  speed  to  2. lx,  and  sits  back  to  view  and  write 
again.  In  about  another  eight  minutes  she  slowed  down  the  presentation  noticeably  to  1.7  times  normal  speed 
for  about  1 minute,  after  which  she  accelerated  back  up  to  1.9x.  Never  did  she  stop  or  replay  sections.  When 
asked  what  motivated  her  to  slow  down  the  audio,  she  said  that  the  content  was  complicated,  and  she  wasn’t 
getting  it — so  she  slowed  down.  This  is  a powerful  example  of  using  VSP  as  a self -regulated  learning  practice. 
Her  lack  of  distraction  was  especially  noteworthy  to  me  since  at  one  point  the  lab  TA  approached  me  and 
attempted  persistently  to  engage  me  in  conversation  about  what  I was  doing.  I tried  without  success  for  what 
seemed  like  minutes  to  communicate  that  I was  busy  and  didn’t  want  to  chat.  In  interviews  Laura  recalled  the 
distraction,  but  ignored  it.  Laura  and  I were  both  wearing  headphones. 

It’s  an  interesting  idea  that  the  use  of  headphones  by  all  the  participants,  except  those  studying  their 
materials  at  home,  helped  them  manage  and  regulate  their  attention.  Participants  never  said  as  much,  but 
common  sense  would  suggest  it  did  help  them  concentrate  and  minimize  distractions  particularly  in  noisy  lab 
setting  like  the  accounting  lab  where  TAs  consulted  with  students  and  study  groups  met  regularly  and  talked  out 
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loud. 


Trisha  also  started  her  lesson  out  at  2. Ox.  She  yawned  repeatedly  during  the  lessons.  After  one  big 
yawn,  she  reached  over  and  tried  unsuccessfully  to  accelerate  the  presentation  above  2. Ox,  but  the  control  was 
maxed  out.  I chuckled  inside.  Since  I was  listening  simultaneously  with  her,  I knew  what  she  was  going 
through.  Having  not  even  had  the  background  of  this  lesson,  I easily  understood  the  “common  sense”  material 
and  was  ready  to  pick  up  the  pace  and  wished  (as  did  she  apparently)  that  the  presentation  could  have  been 
accelerated  at  that  point.  Her  particular  computer  configuration  did  not  allow  higher  speeds  than  2. Ox.  Rather 
than  skip  ahead  and  risk  missing  something,  Trisha  relied  strictly  on  VSP  and  acceleration  to  pick  up  the  pace. 
When  she  ran  into  a more  challenging  section,  she,  like  Laura  slowed  down  to  1.8x  for  about  4 minutes.  Her 
“slow”  period  in  contrast  to  Laura,  was  interrupted  with  numerous  short  section  replayings — of  course  still  at 
the  relatively  high  speed  of  1.8x. 

The  use  of  variable  speed  playback  is  not  an  isolated  event.  It  appeared  to  significantly  impact  they 
way  other  more  typical  media  playback  controls  are  used  such  as  “Pause”,  “Rewind”  and  “Fast  Forward”  and 
has  arguably  spawned  new  controls  such  as  “Jump  Back  x”  [seconds],  Todd  also  employed  VSP  in  a unique 
way.  He  chooses  to  not  adjust  speeds  during  lessons  s ometimes  even  during  the  boring  parts,  instructor  stories, 
or  content  he’s  familiar  with.  “During  slow  times,  I’ll  get  up  and  get  some  other  things  done.,  .make  me  a 
sandwich  and  stuff.”  He  said.  “I  like  to  keep  the  sound  running  so  I don’t  miss  stuff,  but  can  still  get  other 
things  done  until  I get  to  new  material,  and  then  I come  back.”  He  even  described  slowing  lessons  down  a bit 
further,  so  he  could  get  more  other  tasks  done  at  the  same  time. 

When  I posed  the  question  about  what  got  in  the  way  of  her  learning  in  this  course,  Laura  stated 
emphatically  “The  instructor’s  examples!  I think  he  waaaay  over-explains  things,  way!  And  it  bugs  me  cuz  I 
still  have  to  go  through  it.”  She  also  felt  the  need  to  accelerate  through  materials  rather  than  skip  ahead.  As 
alluded  to  earlier,  this  fear  of  skipping  ahead  is  probably  related  to  the  medium  of  video.  Video  cannot  be 
skimmed  in  the  same  way  or  as  efficiently  as  text.  The  use  of  VSP  to  accelerate  presentations,  likely  acts  as  a 
speed-reading  tactic  for  learners. 


Discussion 

Overall,  course  control  affordances  seem  to  facilitate  SRL.  All  my  participants  were  ahead  of  schedule 
in  their  course,  rather  than  procrastinating.  They  hadn’t  painted  themselves  into  a corner — forcing  themselves 
to  go  faster  than  they  should  have,  just  to  meet  course  deadlines --although  recall  that  a TA  had  mentioned  that 
some  class  members  did  fall  prey  to  that  scenario.  Students  were  motivated  by  the  flexibility  of  the  course.  Its 
asynchronous  nature  helped  them  manage  their  study  time  both  in  this  class  as  well  as  in  their  other,  less 
flexible,  classes.  Jack’s  words  represent  the  feelings  of  all  the  participants  in  this  regard: 

I like  being  able  to  do  it  on  my  own  time.  I’m  able  to  listen  to  the  cds  and  what  not,  and 
also  you  can  get  ahead.  You  can  kinda  plan  your  weeks  out. ..If  you  have  a lot  of 
homework  in  your  other  classes  one  week,  you  can  look  ahead — and  get  ahead  in  the 
accounting  lessons,  and  if  that  week  gets  too  hectic  for  ya,  you  don't  have  to  worry  about 
it. 

An  interesting  aspect  of  student’s  self-regulation  is  that  despite  their  limited  time,  participants  will 
wind  up  viewing  far  more  material  than  they  ever  would  have  received  in  the  face-to-face  class.  They  choose  to 
view  everything.  They  view  all  the  remedial  lessons,  and  helps  that  were  designed  for  struggling  students. 

Most  participants  believed  that  because  they  accelerated,  they  must  be  saving  time.  There  were  approximately 
25  hours  of  additional  instruction  recorded  for  this  course  above  and  beyond  what  a student  would  have 
encountered  in  the  face -to  face  version  of  the  course.  This  fact,  combined  with  the  amount  of  replayed 
segments  that  I observed,  was  not  likely  compensated  for  by  student  acceleration  rates. 

The  course  quiz  structure  was  not  generally  conducive  to  self-regulated  learning,  making  it  difficult  to 
be  learning  for  the  right  reasons.  Perhaps  out  of  necessity  in  such  a large  class,  test  scores  are  the  primary  form 
of  feedback  continually  being  given  to  students . This  is  in  contrast  to  qualitative  forms  of  feedback  that  might 
prompt  deeper  self-reflection  in  learners  (Corno  & Randi,  1999).  Yet,  here  too,  students  seem  to  be  taking 
responsibility  for  their  learning  and  not  just  studying  to  perform  well  on  the  quizzes.  Elliott  & Dweck  (1988) 
found  in  a study  with  5th  graders  that  when  children  using  performance  goals  (i.e.  must  score  high  on  quiz), 
failure  and  challenges  are  more  likely  to  provoke  a helpless  response.  But  when  children  were  instead  focused 
on  learning  goals,  failure  and  challenges  were  more  likely  to  “provoke  continued  effort”  (p.  17).  It  would  appear 
that  in  learning  environments  such  as  this  video-centric  accounting  course,  students  that  scored  low  seemed  to 
feel  like  they  just  needed  to  study  harder,  slow  the  video  down  or  study  more  effectively.  One  TA  spoke  to  this 
issue  when  he  described  helping  people  in  the  accounting  lab: 
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A lot  of  students  will  come  to  me  and  say,  ‘hey  I didn’t  get  any  of  this”,  and  I’ll  ask, 

‘what  speed  did  you  watch  it  at?’  The  majority  of  the  time  they  listened  to  it  at  an 
accelerated  pace,  so  I usually  tell  them  to  go  back  and  listen  to  it  again,  slower. 

Rather  than  blame  their  intelligence,  abilities  or  the  teacher — the  course  seems  to  be  structured  to 
support  self-regulatory  practices  and  encourages  s tudents  to  take  responsibility  for  both  their  learning  as  well  as 
for  the  study  strategies  they  employ.  This  is  perhaps  due  to  the  ‘‘independent  study”  nature  of  the  course  and 
the  levels  of  user  control  built  into  the  course  interface.  As  the  designer  of  the  technological  aspects  of  the 
course  (media  and  interface,  but  not  pacing,  assignments,  course  schedule  or  syllabus)  it  was  my  intent  to  build 
in  a great  deal  of  flexibility  into  a medium  (video)  not  commonly  known  for  its  user-control  affordances.  As 
Kozma  (1991)  states,  students  benefit  most  fromtechnology  based  learning  environments  when  the  capabilities 
of  the  environment  are  used  to  perform  or  model  certain  cognitive  operations  that  are  important  to  the  learning 
task  but  that  the  learners  cannot,  or  do  not,  perform  themselves.  Although  numerous  themes  emerged  from  the 
data  along  these  lines,  three  central  themes  are  addressed  that  help  describe  how  students  used  the  playback 
controls  of  the  learning  environment  to  regulate  their  learning.  Each  theme  is  discussed  below. 

Theme  One 

Students  adjusted  playback  speeds  in  an  effort  to  influence  their  attentiveness  and  concentration. 

Self- monitoring  and  regulation  were  evident  as  students  purposefully  adjusted  playback  speeds  in  an  effort  to 
regulate  their  attentiveness  and  concentration.  Students  reported  less  mind  wandering  at  higher  speeds  and  that 
it  actually  reduced  the  number  of  times  to  rewind  and  repeat  phrases  or  thoughts.  Speed  adjustments  played  an 
early  role  in  the  regulatory  behavior  for  all  the  participants  as  they  reported  more  frequently  adjusting  the  speed 
of  the  lectures  in  the  first  few  weeks  of  the  semester,  until  they  found  their  ‘‘sweet  spot”.  This  “sweet  spot” 
tended  to  gradually  increase  over  time  as  they  got  used  to  it. 

Theme  Two 

Once  comfortable  speeds  were  identified,  fewer  speed  adjustments  were  made  and  other  playback 
control  affordances  such  as  repetition,  became  more  dominant.  It  was  clear  during  student  observations  and 
interviews,  that  when  students  felt  they  missed  particular  content,  or  were  not  comprehending  concepts,  they 
chose  more  often  to  repeat-replay- -a  few  lines  or  whole  segment  rather  than  adjust  playback  speed  to  slower 
rates.  A student  noted  that  “It’s  just  easier  to  jump  back  a bit  if  my  room  mate  distracts  me.,  .and  if  I’m  not 
understanding  a section,  I just  understand  it  better  if  I play  the  section  again.  I usually  always  get  it  the  second 
time.”  When  students  did  slow  down  to  aid  in  comprehension,  adjustments  were  relatively  minor  in  all  but  one 
case  where  the  student  replayed  course  materials  at  real-time  or  normal  speed. 

Theme  Three 

Students  choose  to  view  all  course  materials — including  “remedial”  content. 

In  an  interesting  twist  with  regard  to  self-regulation,  students  reported  viewing  all  lesson  content — including 
additional  or  remedial  instruction  designed  for  struggling  students.  This  was  likely  unnecessary  for  some 
students,  but  because  they  were  accelerating  lessons,  there  seemed  to  be  a perception  of  time  saved,  which 
justified  viewing  more  material.  There  were  approximately  25  hours  of  additional  instruction  recorded  for  this 
course,  above  and  beyond  what  a student  would  have  encountered  in  the  face-to  face  version  of  the  course. 

This  fact,  combined  with  the  amount  of  replayed  segments  observed,  was  not  likely  compensated  for  by  student 
acceleration  rates.  “Multitasking  also  helped  save  time”  said  another  student.  “Rather  than  skip  content  I am 
familiar  with,  I’ll  get  up  and  get  some  other  things  done . . .make  me  a sandwich  and  stuff.  I like  to  keep  the 
sound  running  so  I don’t  miss  stuff,  but  can  still  get  other  things  done  until  I get  to  new  material,  and  then  I 
come  back.” 


Conclusion 

Learning  environments  should  be  designed  such  that  they  either  promote  learner  self-regulation  or 
make  up  for  a lack  of  it  (McCombs  & Marizano,  1990).  Additional  research  is  needed  to  explore  the 
relationship  between  control  affordances  and  self-regulated  learning,  but  it  is  clear  that  some  SRL  behaviors 
cannot  occur  without  a certain  level  of  learner  autonomy  (Zimmerman  and  Schunk,  2001)  and  control  over  the 
video  medium.  Of  the  available  micro  controls,  variable  speed  playback  appeared  to  play  a central  regulatory 
role  with  these  learners  in  this  multimedia  accounting  course.  VSP  was  pervasive  in  so  many  of  the 
participant’s  SRL  strategies,  influencing  the  way  they  attended  to,  and  interacted  with  their  course  materials.  Its 
influence  even  carried  over  into  their  traditional  face-to-face  lecture  courses  where  some  students  lamented  not 
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having  the  ability  to  accelerate  and  replay  all  their  university  instructors  in  like  manner. 

If  learning  performance  is  indeed  determined,  in  large  measure,  by  learners’  self-regulation  ability,  it 
should  not  be  assumed  that  all  students  are  equally  skilled  or  experienced  or  consistent  in  their  self-regulating 
behavior.  The  findings  seemed  to  support  the  notion  or  hypothesis  that  explicitly  sharing  of  effective  SRL 
strategies  with  students  would  help  foster  practices  that  facilitate  learning  and  discourage  practices  that  may 
hinder  it--such  as  using  VSP  to  simply  race  through  materials  to  meet  course  deadlines. 

Able  self-regulating  learners  purposefully  monitor  and  guide  their  own  learning  while  learners  less 
skilled  at  self -regulating,  are  more  motivationally  and  metacognitively  passive  in  their  reception  of  instruction 
(McManus,  1998).  But  not  all  SRL  strategies  are  equally  effective.  Traditional,  perhaps  more  macro,  strategies 
outlined  by  Zimmerman  & Schunk  (1989),  Winne  & Stockley  (1998),  Ley  & Young  (2001)  and  Pintrich  (1995) 
are  likely  more  critical  in  developing  sustainable  and  broadly  applicable  SRL  habits,  than  are  the  micro  SRL 
processes  focused  on  in  this  study.  Nevertheless,  attempting  to  better  understand  the  supporting  role  of 
technology  in  the  development  and  facilitation  of  SRL  strategies  in  video-centric  environments  is  deemed  to  be 
a valuable  pursuit  for  designers  and  continued  research  is  encouraged  that  attempts  to  validate  and  test  the 
hypotheses  generated  by  this  study. 
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Abstract 

Hie  results  of  a citation  pattern  analysis  on  the  journal  Educational  Technology  Research  and 
Development  (ETR&D)  for  the  period  1990-99  are  presented.  Reference  lists  for  the  260  articles  published 
during  that  period  were  compiled  along  with  over  1,600  citations  of  those  articles  in  other  social  science 
journals.  Breakdowns  of  most  commonly  cited  references  during  that  period  are  presented  along  with  an 
analysis  of  the  journals  more  cited  by  and  most  citing  of  ETR&D. 

Introduction 

I want  to  see  the  book  you  stole  there  after  reading  it,  to  keep  others  from  reading  it,  and  you  hid  it 
here,  protecting  it  cleverly,  and  you  did  not  destroy  it  because  a man  like  you  does  not  destroy  a book,  but 

simply  guards  it  and  makes  sure  no  one  touches  it.  I want  to  see , the  book  that  everyone  has 

believed  lost  or  never  written,  and  of  which  you  hold  perhaps  the  only  copy. 

Brother  William  of  Baskerville 
In  Umberto  Eco’s  The  Name  of  the  Rose 

There  is  a growing  mountain  of  research.  But  there  is  increased  evidence  that  we  are  being 
bogged  down  today  as  specialization  extends.  The  investigator  is  staggered  by  the  findings 
and  conclusions  of  thousands  of  other  workers — conclusions  which  he  cannot  find  time  to 
grasp,  much  less  reme  mber,  as  they  appear. . . A record,  if  it  is  to  be  useful  to  science,  must  be 
continuously  extended,  it  must  be  stored,  and  above  all  it  must  be  consulted. 

Vannevar  Bush 

“As  We  May  Think”  in  Atlantic  Monthly,  1945 

Social  science  research  is  itself  a social  endeavor.  Although  media  representations  often  depict 
researchers  as  solitary  figures  struggling  independently  to  discover  “truth,”  productive  research  is  conducted 
within  the  framework  of  a community.  “Research  is  complete  only  when  the  results  are  shared  with  the 
scientific  community.”  (APA,  2001,  p.  3).  In  addition  to  collaboration  with  other  researchers  and  the  need  to 
build  upon  (and  sometimes  tear  down)  the  work  of  predecessors,  the  highest  goals  of  research  are  defined  by 
peer  review.  This  examination  of  works  by  a qualified  audience  is  required  to  ensure  that  conference 
presentations  and  published  articles  represent  the  best  that  the  field  has  to  offer. 

The  examination  of  the  artifacts  of  this  process  inform  us  of  not  only  the  perceptions  and  viewpoints  of 
the  individual  authors,  but  also  of  these  “peers”  who  serve  as  science’s  gatekeepers.  A number  of  researchers 
have  exa  mined  the  content  of  academic  journals  in  order  to  capture  trends  or  patterns  of  research  behavior. 

Klein  (1997)  exa  mined  a nine-year  period  of  publication  of  the  development  section  of  Educational  Technology 
Research  and  Development  (ETR&D).  His  content  analysis  indicated  the  “ID  for  Computer  Technologies”  and 
“Instructional  Design  & Development”  were  the  most  popular  article  topics  and  that  almost  half  of  all  articles 
were  descriptions  of  different  activities  with  little  or  no  supporting  data.  In  a separate  analysis,  Klein  (2002) 
performed  a similar  analysis  on  four  years  of  research  published  in  Performance  Improvement  Quarterly.  In  that 
analysis,  about  one-third  of  the  articles  contained  empirical  research.  This  follow-up  research  not  only 
demonstrates  that  similar  methodology  can  be  applied  to  different  journals  in  the  field,  but  also  that  different 
journals  in  the  same  field  may  have  uniquely  differing  publication  patterns.  In  terms  of  looking  for  similarities, 
ETR&D  was  one  of  four  journals  examined  by  a group  of  researchers  from  the  University  of  Kentucky  (Anglin, 
Cain,  Whitehouse,  Cunningham,  Newcomer,  and  Cunningham,  2003).  The  other  three  journals  were  the  British 
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Journal  of  Educational  Technology,  Educational  Technology,  and  the  Journal  of  Research  on  Technology  in 
Education.  During  the  five  year  period  examined  (1997-2001),  they  identified  the  ERIC  descriptors  “the 
educational  process:  classroom  perspectives”  and  “the  educational  process:  societal  perspectives”  as  the  most 
used  in  indexing  article  content.  Driscoll  & Lee  (2003)  identified  trends  in  distance  education  by  analyzing 
article  content  for  four  selected  journals  over  a five-year  period.  Among  other  results,  they  found  that 
developmental  research  was  rare  and  that  case  studies  were  the  most  prominent  type  of  article. 

Citation  analysis  in  particular  provides  an  opportunity  for  authors  to  understand  the  influence  and 
influencers  of  their  work.  At  the  level  of  an  individual  article,  an  author  cites  others  so  that  the  reader  can  see 
the  social  construction  of  knowledge.  Publication  represents  successfully  completing  the  peer  review  process, 
but  citation  of  one’s  work  is  a greater  indicator  of  the  influence  of  the  work.  Although  those  who  cite  may  agree 
or  disagree  with  an  author’s  work,  citation  denotes  that  the  previous  work  was  worthy  of  discussion.  Citation 
analysis  is  less  common  than  other  types  of  research  on  publication  patterns,  but  should  become  more 
accessible  with  improvements  in  on-line  resources  and  databases.  For  example,  Frisby  (1998)  examined  citation 
patterns  in  five  years  of  seven  major  journals  in  school  psychology.  An  interesting  aspect  of  this  research  was 
his  definition  of  self-citation  within  a journal  as  “within -journal  inbreeding,”  a negative  characteristic  from  his 
point  of  view.  Another  interesting  example  of  citation  analysis,  was  conducted  by  Oppenheim  and  Smith 
(2001).  They  focused  on  the  citation  patterns  of  final  year  students  in  an  information  science  department.  In 
direct  contrast  to  “refereed”  publications,  their  analysis  indicated  an  increasing  trend  of  students  citing  Internet 
resources  rather  than  more  traditional  sources.  Creamer  (1998)  examined  citation  patterns  as  a possible  measure 
of  faculty  publication  productivity.  She  reported  that  authors  tend  to  cite  more  heavily  other  authors  of  the  same 
gender.  However,  she  suggested  that  this  may  be  reflective  of  homogeneous  gender  informal  professional 
networks  rather  than  conscious  bias. 


Limitations 

The  current  study  is  decidedly  descriptive  in  nature.  Also,  the  researchers  believe  that  citations  are  a 
measure  of  importance,  not  the  measure  of  importance.  It  must  be  added  that  not  all  citations  are  equal,  their 
relative  importance  varies  and  a citation  may  be  made  to  show  agreement  or  disagreement  with  a particular 
point.  However,  it  is  suggested  that  patterns  of  citation  data  are  informative,  certainly  as  much  as  course  surveys 
or  opinion  polls. 


Method 

For  the  current  study,  the  journal  Educational  Technology  Research  and  Development  (ETRD)  was 
examined  over  a ten-year  publication  period,  1990-99.  This  journal  was  selected  because  of  its  reputation  in  the 
field.  For  example,  Maushak,  Price,  and  Wang  (2000)  conducted  a survey  of  85  faculty  members  in  the  field  of 
educational  technology.  According  to  their  analysis,  ETR&D  was  the  overwhelming  choice  as  the  top  journal  in 
the  field.  During  this  period,  40  issues  were  published  containing  260  journal  articles.  For  the  purpose  of  this 
study,  only  articles  that  were  part  of  the  Research  or  Development  sections  were  considered. 

The  tables  of  content  for  the  issues  were  photocopied  and  used  to  uniquely  identify  each  article  and 
record  relevant  data  into  an  Excel  worksheet.  One  research  , using  the  online  version  of  the  Social  Science 
Citation  Index  (SSCI),  performed  a search  on  each  article  and  recorded  information  on  each  citing  article  into  a 
second  worksheet  (N=l,553). 

Another  researcher  created  a photocopied  record  of  the  reference  lists  for  the  260  articles.  Due  to  the 
large  number  of  references  (N  = 10,055),  a number  of  individuals  manually  entered  this  reference  data  into  a 
third  worksheet.  After  these  references  were  coded,  one  individual  reviewed  each  entry  in  comparison  with  the 
printed  record  to  ensure  consistency  and  correct  minor  errors. 

All  analysis  was  conducted  manually  and/or  using  the  functionality  of  Excel  (for  example,  to  count  or 
aggregate  results).  During  analysis,  the  photocopied  indices  and  reference  lists  were  consulted  to  clarify  and/or 
correct  entries. 

As  with  any  intensive  data  entry  endeavor,  errors  were  made  in  collecting  and  recording  the  data.  All 
attempts  were  made  to  correct  identified  errors  and  the  impact  on  aggregrated  results  should  be  minimal. 

The  limited  journal  scope  (only  ETR&D)  and  publication  period  (10  years)  were  selected  arbitrarily  by 
the  researchers.  Due  to  the  descriptive  nature  of  the  analysis,  any  inference  or  interpretation  must  be  made 
cautiously. 
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Results 

Of  the  260  core  journal  articles,  94  (36.15%)  were  authored  by  one  individual.  Two  authors  were 
involved  in  the  writing  of  98  articles  (37.69%).  Sixty-eight  articles  (26.15%)  were  written  by  three  or  more 
authors  with  nine  authors  being  the  most  for  any  article. 

The  researchers  attempted  to  establish  the  gender  of  the  various  authors.  This  was  largely 
idiosyncratic.  A large  number  of  authors  were  identified  via  professional  contact  or  a gender-specific  name. 
Others  were  identified  by  locating  biographical  information  on  the  Internet.  For  the  94  articles  written  by  one 
author,  58%  were  male;  33%  were  female;  and  9%  were  unknown.  ETR&D  uses  a blind  review  process  in 
which  the  identity  and  gender  of  the  author  are  unknown.  Without  additional  knowledge  of  submissions  by 
gender  or  general  gender  makeup  of  the  field,  it  is  impossible  to  determine  specific  causes  for  the  lower 
publication  numbers  for  females.  For  the  166  articles  of  multiple  authorship,  44%  included  at  least  one  male 
and  one  female;  27%  had  exclusively  male  authors;  9%  had  exclusively  female  authors;  and  20%  could  not  be 
determined.  The  large  number  of  undetermined  makes  analysis  difficult.  The  large  number  of  mixed  gender 
efforts  is  encouraging.  The  large  discrepancy  between  male -only  (27%)  and  female -only  (9%)  efforts  may  again 
reflect  gender  distribution  in  the  field  or  preferences  in  informal  networking. 

Author  productivity  was  classified  according  to  order  of  authorship  (see  Table  1).  Those  that  had  the 
largest  number  of  first  author  articles  were  arbitrarily  labeled  “The  Prolific”  to  represent  the  primary  importance 
given  to  first  authors  in  academic  communities.  Those  with  the  largest  number  of  second  author  articles  were 
labeled  “The  Mentors,”  suggesting  their  role  in  helping  others  with  attaining  first  author  credits  in  this  journal. 
Lastly,  those  that  had  the  greatest  number  of  authorships  of  third  or  later  priority  were  labeled  “The 
Collaborators,”  tenuously  identifying  them  for  a more  minor  yet  continuing  role  in  assisting  in  authorship. 

Table  1.  Author  Productivity 

The  Prolific  (First  Authors)  Number 

Mable  B.  Kinzie 
Lloyd  Rieber 
Martin  Tessmer 

The  Mentors  (Second  Authors)  Number 

Michael  J.  Hannafin 
Howard  J.  Sullivan 
James  D.  Klein 

The  Collaborators  (Third... Authors)  Number 

Gary  R.  Morrison 
John  F.  Wedman 
John  Bransford 


of  Articles 

6 

6 

6 

of  Articles 

8 

7 

5 

of  Articles 

4 

4 

3 


Of  the  10,055  references  cited  by  the  core  articles,  53%  were  to  journal  articles;  36%  were  to  books; 
and  11%  were  classified  as  other  (e.g.  websites,  technical  reports,  dissertations,  etc.).  The  most  cited  journals 
and  most  cited  journal  article  authors  are  presented  in  Table  2. 


Table  2.  Most  Cited  Journals  and  Journal  Article  Authors 


Journals 

ETR&D 

Educational  Technology 
Journal  of  Educational  Psychology 
Review  of  Educational  Research 
Educational  Researcher 
Journal  Article  Authors 
David  H.  Jonassen 
Michael  J.  Hannafin 
Gavriel  Salomon 
Robert  D.  Tennyson 
Richard  E.  Mayer 
Steve  M.  Ross 


Number  of  References 

494  (9.40%) 

351  (6.68%) 

331  (6.30%) 

177  (3.37%) 

163  (3.10%) 
Number  of  References 
76(1.45%) 
64(1.22%) 
60(1.14%) 
59(1.12%) 
58(1.10%) 
58(1.10%) 
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The  references  to  books  were  further  subdivided  into  edited  and  non-edited  books.  The  most  cited  for 
both  categories  are  shown  in  Table  3.  For  this  analysis,  multiple  editions  of  a work  were  counted  together  (the 
year  of  the  earliest  edition  is  provided  in  the  table). 


Table  3 . Most  Cited  Edited  and  Non  -edited  Books 

Number  of 

Edited  Books 

Editor 

References 

Instructional  Design  Theories  and 
Models:  An  Overview  of  Their 

C.M.  Reigeluth 

55 

Current  Status  (1983) 
Instructional  Technology: 

R.M.  Gagne 

34 

Foundations  (1987) 

Handbook  of  Research  on  Educational 

D.H.  Jonassen 

33 

Communications  and  Technology 
(1996) 

Instructional  Technology:  Past, 
Present,  and  Future  (1991) 

G.J.  Anglin 

31 

Knowing,  Learning,  and  Instruction: 
Essays  in  Honor  of  Robert  Glaser 

L.B.  Resnick 

28 

(1989) 

Number  of 

Non-edited  Books 

Authors 

References 

The  Systematic  Design  of  Instruction 

W.  Dick 

40 

(1978) 

L.  Carey 

The  Conditions  of  Learning  and 
Theory  of  Instruction 

R.M.  Gagne 

39 

Principles  of  Instructional  Design 

R.M.  Gagne 

35 

(1974) 

Cooperative  Learning:  Theory, 

R.E.  Slavin 

14 

Research,  and  Practice  (1983) 

Mind  in  Society:  The  Development  of 
Higher  Psychological  Processes 

L.S.  Vygotsky 

13 

(1978) 

Teaching  for  Competence  (1983) 

H.  Sullivan 

12 

The  260  core  articles  were  entered  into  the  Social  Science  Citation  Index  (SSCI)  in  late  2003  to 
determine  in  which  journals  these  articles  were  cited.  Table  4 displays  the  most  citing  journals  and  the  core 
articles  that  were  the  most  cited.  Because  the  core  articles  were  published  over  a period  of  10  years  and  the 
SSCI  data  represented  a snapshot  of  all  citations  to  that  point,  it  was  determined  that  average  citations  since 
year  of  publication  would  provide  the  most  unbiased  measure  of  influence. 


Table  4.  Most  Citing  Journals  and  Most  Cited  Articles 


Number  of 

Citing  Journals  Citations 

ETR&D  475  (30.59%) 

Journal  of  Educational  Computing  Research  100  (6.44%) 

Computers  in  Human  Behavior  59  (3.80%) 

Instructional  Science  51(3.28%) 

British  Journal  of  Educational  Technology  50  (3.22%) 

Most  Cited  Articles  Citations/Yr 

Media  Will  Never  Influence  Learning  by  R.E.  Clark  7.78 

Objectivism  versus  Constructivism:  Do  We  Need  a New  4.83 

Philosophical  Paradigm  by  D.H.  Jonassen 

Instructional  Design  for  Situated  Learning  by  M.F.  Young  4.40 
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Will  Media  Influence  Learning?  Reframing  the  Debate  by  4.22 

R.B.  Kozma 


Conclusion 

These  preliminary  results  provide  guidance  for  professionals  and  students  in  the  field  as  to  what  some 
consider  the  influential  works  and  individuals  with  which  one  should  be  familiar.  The  methodology  used 
advances  what  has  previously  been  published  in  the  form  of  course/program  surveys  and  opinion  polls. 

A more  complete  presentation  is  currently  being  developed  for  eventual  publication.  The  research  team 
has  also  begun  to  collect  data  on  related  journals  during  the  same  time  period.  This  will  allow  comparison  and 
triangulation  with  this  dataset.  Also,  data  collection  has  begun  with  ETR&D  for  the  next  five-year  period  (2000- 
2004).  This  will  allow  chronological  comparison  with  the  present  dataset.  Finally,  the  improvement  of  online 
tools  continues  to  make  the  process  more  manageable.  As  journals  and  databases  add  functionality  and  online 
content,  more  advanced  analysis  will  become  increasingly  automated  and  informative. 
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Abstract 

Four  fifth  grade  classrooms  embarked  on  a modified  ubiquitous  computing  initiative  in  Fall  2003. 

Two  15-computer  wireless  laptop  carts  were  shared  among  the  four  classrooms  in  an  effort  to  integrate 
technology  across  the  curriculum  and  affect  change  in  student  learning  and  teacher  pedagogy.  This  initiative — 
in  contrast  to  other  1:1  programs  and  stationary  labs — offers  public  schools  alternatives  to  budget  constraints 
and  instructional  space  overhead.  Results  indicate  positive  teacher  technology  competence  and  confidence,  as 
well  as  instructional  strategies  that  were  student  centered  made  meaningful  uses  of  technology.  Teacher 
pedagogical  knowledge,  technological  knowledge  and  a supportive  culture  seem  to  be  strong  indicators  for 
impacting  technology  integration  in  this  context. 

As  access  to  computer  technologies  continues  to  increase  (National  Center  for  Educational  Statistics, 
2001),  there  has  also  been  a movement  to  decrease  the  computer  to  student  ratio,  as  well.  Windschitl  and  Sahl 
(2002)  report  “more  than  a thousand  schools  nationwide  have  committed  themselves  to  some  form  of  laptop 
computer  initiative”  (p.  165).  Goals  of  these  types  of  initiatives  vary,  but  include  increased  student 
achievement  and  learning,  increased  home  and  school  interaction  and  increased  technology  access  for  low 
income  families  (Penuel  et  al.,  2002). 

A number  of  evaluations  and  case  studies  have  documented  mixed  results  with  ubiquitous  computing 
approaches  (e.g.,  Edwards,  2003;  Hill,  Reeves,  Grant,  Wang,  & Hans,  in  press;  Lowther,  Ross,  & Morrison, 
2003;  Mowen,  2003;  Rockman  et  al.,  1997,  2000).  Successes  have  included  improved  student  achievements, 
broader  access  and  equity  for  students,  increased  communications  among  faculty,  administrators,  students  and 
parents  and  reports  of  reductions  in  absenteeism  and  school  dropout  rates.  However,  teachers  and  students  alike 
struggled  with  managing  learning  issues,  such  as  time  on  task;  pedagogical  issues,  such  as  transitioning  to 
student-centered  learning;  and  classroom  management  issues,  such  as  monitoring  student  distractions  with 
email,  the  Internet  and  gaming. 

Advocating  a one-to-one  student  to  computer  ratio  is  appealing,  and  the  goals  are  admirable.  A similar 
statewide  initiative  (Bickford,  Tharp,  McFarling,  & Beglau,  2002)  has  attempted  to  compromise  on  the  student 
to  computer  ratio  at  two  students  to  everyone  one  computer.  However,  this  program  has  also  been  challenged 
with  lack  of  change  in  teacher  practice.  And  more  recently,  funding  limits  have  placed  the  onus  of  support  on 
the  individual  school  districts. 

The  funding  for  such  innovative  programs  cannot  be  discounted.  Supplying  every  student  and  teacher 
with  an  Internet-capable  computer  is  a substantial  capital  commitment  for  school  budgets.  One  principal  admits 
his  middle  school  has  “invested  well  over  $1  million  in  laptop  technology... since  2000”  (Mowen,  2003, 
Introduction  section,  para.  3).  For  this  reason  many  early  laptop  initiatives  were  implemented  in  private  and 
parochial  schools  (e.g..  Hill  et  al.,  in  press;  Newhouse,  2001;  Rockman  et  al.,  1997). 

As  an  alternative  to  a one-to-one  initiative  for  a select  group  of  students,  some  schools  have  purchased 
mobile  laptop  carts.  These  carts  of  5 to  25  mobile  computers  are  typically  wireless  and  can  be  wheeled  from 
classroom  to  classroom  as  needed.  Schools  have  used  this  model  to  promote  collaboration  among  students  and 
aid  in  transitioning  among  groups  of  students  and  in  classroom  settings  (e.g.,  Gwaltney,  2003).  In  addition, 
these  mobile  carts  have  also  offered  an  alternative  to  committing  instructional  space  to  computer  laboratories. 

Little  research  has  been  reported  about  these  mobile  laboratories.  This  small  evaluation  study  presents 
the  findings  of  one  school’s  experience,  identified  with  the  pseudonym  Green  River  Elementary,  with  using 
mobile  laptop  carts  to  affect  change  in  teacher  practice  and  student  learning.  The  laptop  program  evaluation 
was  structured  around  five  primary  research  questions  that  focused  on  classroom  practices,  degree  and  type  of 
technology  use,  academically  focused  time,  student  engagement,  teacher  technology  skills,  teacher  attitudes 
toward  technology,  as  well  as  student  and  teacher  reactions  to  the  program.  The  research  questions  were; 
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1. 


In  what  ways  has  the  effectiveness  of  instruction  through  the  use  of  student  laptop  computer  been 
impacted? 

2.  To  what  degree  and  in  what  ways  have  teachers  integrated  technology  with  classroom  instruction? 

3.  To  what  degree  do  teachers  use  methodologies  that  stress  higher-order  learning  and  student -centered 
learning  activities? 

4.  To  what  degree  has  the  laptop  program  impacted  teacher  attitudes  toward  technology? 

Design 

The  evaluation  design  was  based  on  both  quantitative  and  qualitative  data  collected  from  classroom 
observations,  teacher  surveys,  and  focus  groups  with  teachers  and  students.  The  four  fifth  grade  teachers  and 
their  intact  classrooms  at  Green  River  Elementary  participated  in  the  evaluation. 

Context 

Green  River  Elementary,  serving  grades  K-8,  was  situated  in  a suburban  city  outside  a large  urban  city 
in  the  southeast  United  States.  The  laptop  program  was  a pilot  project  designed  to  determine  the  impact  of 
changing  the  ways  students  learn  and  teachers  instruct  in  a technology -enhanced  learning  environment.  The 
original  concept  included  a laptop  for  each  student  in  fifth  grade,  replicating  Rockman  et  al.’s  (1997) 
concentrated  implementation  model,  with  dedicated  technology  integration  training  for  the  fifth  grade  teachers,. 
Unfortunately,  costs  and  lack  of  significant  grant  funding  prohibited  the  implementation  of  the  project  to  this 
extent. 

Instead,  the  context  for  the  laptop  program  and  this  evaluation  consisted  of  four  fifth  grade  classes  in 
which  two  Apple  iBook  wireless  laptop  carts  were  shared  among  the  four  fifth  grade  teachers’  classrooms.  In 
addition,  the  four  teachers  each  received  a personal  Apple  PowerBook  laptop  to  use  during  the  initiative  and 
individually  focused  their  professional  development  opportunities  on  technology-related  training  offered 
through  the  local  school  district.  Each  of  the  fifth-grade  teachers  taught  one  of  the  core  subject  areas,  language 
arts,  math,  science  and  social  studies.  So  every  fifth-grade  student  rotated  through  each  teacher’s  classroom 
during  the  day.  The  number  of  students  per  class  ranged  from  23  to  27. 

Data  Collection 

Five  instruments  and  focus  group  interviews  were  used  to  collect  the  evaluation  data  (three  classroom 
observation  measures,  two  teacher  surveys  and  four  interviews). 

Classroom  Observation  Measures.  Observations  were  made  focusing  on  targeted  classes  (scheduled 
visits)  using  three  instruments.  Descriptive  statistics  were  used  for  analyses.  The  School  Observation  Measure 
(SOM&)  examined  the  frequency  of  usage  of  24  instructional  strategies,  including  traditional  practices  (e.g., 
direct  instruction  and  independent  seatwork)  and  alternative,  predominately  student-centered  methods 
associated  with  educational  reforms  (e.g.,  cooperative  learning,  project-based  learning,  inquiry,  discussion, 
using  technology  as  a learning  tool)  (Ross,  Smith,  & Alberg,  1999).  The  observer  summarized  the  frequency 
with  which  each  of  the  strategies  was  observed  on  a data  summary  form.  The  frequency  is  recorded  via  a 5- 
point  rubric  that  ranges  from  (0)  Not  Observed  to  (4)  Extensively.  Two  global  items  used  three-point  scales 
(low,  moderate,  high)  to  rate,  respectively,  the  use  of  academically  focused  instructional  time  and  degree  of 
student  attention  and  interest.  Targeted  observations  were  conducted  in  this  evaluation  to  examine  classroom 
instruction  during  prearranged  one-hour  sessions  in  which  the  teachers  demonstrated  a prepared  lesson  using 
technology.  Observation  forms  were  completed  every  15  minutes  of  the  lesson  then  were  condensed  on  a 
summary  form.  To  triangulate  the  reliability  of  these  results,  multiple  researchers  observed  class  sessions. 

The  Survey  of  Computer  Use  ( SCU ®)  examined  availability  of  and  student  use  of  technology  and 
software  applications  (Lowther  & Ross,  1999).  Four  primary  types  of  data  were  recorded:  (a)  computer 
capacity  and  currency,  (b)  configuration,  (c)  student  computer  ability  and  (b)  student  activities  while  using 
computers.  Computer  capacity  and  currency  was  defined  as  the  age  and  type  of  computers  available  for  student 
use  and  whether  or  not  Internet  access  was  available.  Configuration  refered  to  the  number  of  students  working 
at  each  computer  (e.g.,  alone,  in  pairs,  in  small  groups).  Student  computer  ability  was  assessed  by  recording  the 
number  of  students  who  were  computer  literate  (e.g.,  easily  used  software  features/menus,  saved  or  printed 
documents)  and  the  number  of  students  who  easily  used  the  keyboard  to  enter  text  or  numerical  information. 
Student  use  of  computers  was  focused  on  the  types  of  computer- mediated  activities,  subject  areas  of  activities, 
and  software  being  used.  The  computer  activities  were  divided  into  three  categories  based  on  the  type  of 
software  tool  (i.e.,  production  tools,  Internet/research  tools,  and  educational  software).  The  final  section  of  the 
SCU  was  an  “overall  rubric’’  designed  to  assess  the  degree  (1:  Low-level  use  of  computers,  2:  Somewhat 
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meaningful,  3:  Meaningful,  4:  Very  meaningful)  to  which  the  activity  reflects  “meaningful  use”  of  computers  as 
a tool  to  enhance  learning. 

Finally,  the  Rubric  for  Student -Centered  Activities  ( RSCA ®)  rated  the  degree  of  learner  engagement 
in  cooperative  learning,  project-based  learning,  higher-level  questioning,  experiential/hands-on  learning,  student 
independent  inquiry/re  search,  student  discussion,  and  students  as  producers  of  knowledge  using  technology 
(Lowther,  Ross,  & Plants,  2000).  These  strategies  reflected  emphasis  on  higher-order  learning  and  attainment  of 
deep  understanding  of  content  and  whether  or  not  technology  was  utilized  as  a component  of  the  strategy.  Such 
learning  outcomes  seemed  consistent  with  those  likely  to  be  engendered  by  well-designed,  real-world  linked 
exercises,  projects,  or  problems  utilizing  technology  as  a learning  tool.  Each  item  included  a two-part  rating 
scale.  The  first  was  a four-point  scale,  ranging  from  (1)  indicating  a very  low  level  of  application  to  (5) 
representing  a high  level  of  application.  The  second  was  a Yes/No  option  to  the  question:  “Was  technology 
used?”  with  space  provided  to  write  a brief  description  of  the  technology  use. 

Surveys.  Two  surveys  were  used  to  obtain  self-perceptions  of  attitudes  and  skills.  The  Teacher 
Technology  Questionnaire  (TTQ)  collected  teacher  perceptions  of  computers  and  technology.  In  the  first 
section,  teachers  rate  their  level  of  agreement  with  20  statements  regarding  five  technology-related  areas:  (a) 
impact  on  classroom  instruction,  (a)  impact  on  students,  (c)  teacher  readiness  to  integrate  technology,  (d) 
overall  support  for  technology  in  the  school  and  (e)  technical  support.  Items  were  rated  with  a five-point 
Likert -type  scale  that  ranges  from  (1)  Strongly  Disagree  to  (5)  Strongly  Agree.  A sixth  section  was  added 
specifically  to  address  perceptions  of  the  laptop  program  also  rated  on  a five-point  scale  from  (1)  Strongly 
Disagree  to  (5)  Strongly  Agree. 

In  addition,  the  Technology  Skills  Assessment  (TSA)  assessed  the  self-perceived  technological  abilities 
of  the  teachers  in  these  areas:  (a)  computer  basics,  (b)  software  basics,  (c)  multimedia  basics,  (d)  Internet  basics, 
(e)  advanced  skills,  (f)  using  technology  for  learning  and  (g)  attitudes  toward  the  laptop  program.  The  survey 
consisted  of  47  items  with  three  levels  (l:Not  at  All,  2:  Somewhat,  3:  Very  Easily).  All  of  the  questions  were 
aligned  to  the  International  Society  for  Technology  in  Education’s  (ISTE)  National  Educational  Technology 
Standards  (NETS). 

Focus  Groups.  Focus  groups  were  conducted  with  all  four  fifth  grade  teachers  and  eight  to  ten  fifth 
grade  students  at  the  beginning  of  the  initiative  in  Fall  2003  and  again  at  the  conclusion  of  the  academic  year  in 
May  2004.  A semi  -structured  interview  protocol  was  used  in  order  to  variation  in  the  order  and  phrasing  of  the 
questions,  as  well  as  probes  to  specific  individuals  (Patton,  1990).  Questions  addressed  three  areas:  (a)  use  of 
laptop  computers,  (b)  expectations  for  the  laptop  program  and  (c)  reservations  about  the  laptop  program. 
Analysis  of  the  data  followed  a general  qualitative  analysis  process  (Cresswell,  1998;  Merriam,  1998).  From 
audio  recordings  and  facilitator  notes,  themes  were  derived  from  the  participants. 

Procedure 

Data  for  this  evaluation  study  were  collected  primarily  in  Spring  2004.  The  SOM,  SCU  and  RSCA 
were  completed  for  each  targeted  observations.  These  consisted  of  prearranged  one-hour  sessions  in  which  the 
fifth  grade  teachers  demonstrated  a prepared  lesson  using  technology.  Observation  forms  were  completed  every 
15  minutes  of  the  lesson.  A total  of  9 visits  across  the  four  fifth  grade  classrooms  were  completed.  The  teacher 
surveys  (TTQ  and  TSA)  were  administered  in  May  2004  prior  to  a focus  group  interview.  Focus  group 
interviews  were  conducted  with  students  and  teachers  in  Fall  2003  and  again  in  May  2004. 

Results 

Below  is  a brief  summary  of  the  results  grouped  by  Classroom  Observation  Results,  Survey  Results 
and  Focus  Group  Results. 

Classroom  Observation  Measures  The  data  for  9 classroom  observations  were  collected  with  SOMs, 
SCUs  and  RSCAs  during  prearranged  sessions  in  which  teachers  implemented  a lesson  using  technology. 
Results  from  each  measure  are  described  in  the  sections  below. 

School  Observation  Measure  (SOM®).  The  SOM  revealed  nine  instructional  strategies  that  were 
observed  during  the  targeted  observations  (N= 9):  (a)  project-based  learning,  (b)  technology  as  a learning  tool  or 
resource,  (c)  teacher  acting  as  coach/facilitator,  (d)  independent  seatwork,  (e)  cooperative/collaborative 
learning,  (f)  independent  inquiry/research  on  the  part  of  the  students,  (g)  higher  level  instructional  feedback  to 
students,  (h)  use  of  higher-level  questioning  strategies  and  (i)  direct  instruction.  These  strategies  were  observed 
during  1 1.1%  of  the  visits  to  100%  of  the  classroom  visits.  Project-based  learning  and  technology  as  a learning 
tool  or  resource  were  observed  during  every  visit  (100%).  Teacher  acting  as  coach/facilitator  and  independent 
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seatwork  were  observed  at  least  50%  of  the  time,  88.9%  and  55.6%  of  the  time  respectively.  Three  more 
instructional  strategies  were  observed  in  at  least  one  third  of  the  visits:  cooperative/collaborative  learning 
(44.4%),  independent  inquiry/research  (44.4%)  and  higher-level  instructional  feedback  (33.3).  Academically 
focused  class  time  and  student  engagement  were  observed  to  be  high  100%  of  the  time. 

Survey  of  Computer  Use  ( SClP ).  In  all  of  the  visits  (N=  9)  (100%),  11  or  more  computers  were 
available  for  student  use  in  the  classrooms.  All  of  the  computers  (100%)  were  observed  to  be  up-to-date,  and 
all  the  computers  (100%)  were  connected  to  the  Internet.  It  is  important  to  note  that  during  two  of  the  pre- 
arranged visits,  the  school’s  internal  network  was  intermittent.  So  while  the  computers  were  capable  of 
accessing  the  intranet  and  Internet,  students  were  unable  to  do  so  consistently.  Student  primarily  worked  alone 
(88.9%)  and  in  pairs  (11.1%).  Computer  literacy  skills  were  observed  to  be  consistently  very  good  (100%),  and 
keyboarding  skills  were  also  very  good  (100%).  Three  student  computer  activities  were  observed  in  at  least 
40%  of  the  classroom  visits:  Internet  browsers  (66.7%),  draw/paint/graphics  (44.4%)  and  electronic 
presentations  (44.4).  Computer  activities  were  observed  in  all  subject  areas.  Productivity  tool  (i.e.  word 
processing,  draw/paint/graphics,  spreadsheets,  etc.)  were  observed  in  language  arts  (33.3%),  mathematics 
(33.3%),  social  studies  (22.2%)  and  science  (11.1%).  Internet/research  tools  were  observed  in  language  arts 
(22.2%),  social  studies  (22.2%),  mathematics  (11.1%)  and  science  (11.1%).  Drill/practice/tutorial  were  the  only 
educational  software  observed  and  only  observed  in  mathematics  (11.1%).  Meaningful  uses  of  computers  were 
extensively  observed  in  over  50%  of  the  visits  (55.6%)  and  very  meaningful  computer  applications  were 
extensively  observed  in  over  30%  of  the  classrooms  (33.3%). 

Rubric  for  Student-Centered  Activities  (RSCA@).  Five  of  the  seven  activities  noted  on  the  RSCA  were 
observed  during  visits  (N= 9):  (a)  project-based  learning  (100%),  (b)  students  as  producers  of  knowledge 
(88.9%),  (c)  cooperative  learning  (44.4%),  (d)  independent  inquiry/research  (44.4%)  and  higher- level 
questioning  strategies  (11.1%).  Notably,  project-based  learning  was  observed  during  all  observations  and 
students  as  producers  of  knowledge  were  observed  during  almost  90%  of  the  visits.  The  most  meaningful 
applications  of  student-centered  activities,  that  is  those  activities  where  somewhat  strong  and  strong 
applications  were  observed  in  at  least  30%  of  the  classroom  visits,  included  cooperative  learning  (44.4%), 
project-based  learning  (77.7%)  and  students  as  producers  of  knowledge  (77.8%).  Technology  was  used  to 
support  three  of  these  strategies:  project-based  learning  (100%),  cooperative  learning  (44.4%)  and  independent 
inquiry/re  search  (44.4%). 

Survey  Results 

Two  surveys  (TTQ  and  TSA)  were  administered  to  the  teachers  prior  to  a focus  group  interview  in 
May  2003.  Results  of  the  two  surveys  are  presented  below. 

Teacher  Technology  Questionnaire  (TTQ).  The  fifth  grade  teachers  (N=  4)  responded  very  positively 
to  the  program.  Mean  scores  for  all  six  sections  were  between  4 (“Agree”)  and  5 (“Strongly  Agree”).  This 
indicates  the  teachers  felt  the  laptop  program  has  had  a positive  impact  on  (a)  classroom  instruction,  (b) 
technology  with  students,  (c)  the  teachers’  readiness  to  integrate  technology,  (d)  the  school  and  district’s  overall 
support  for  technology,  (e)  appropriate  technical  support  and  (f)  a positive  attitude  toward  the  laptop  program. 
Notable  are  the  questions  that  the  teachers  responded  in  unison.  Within  the  section  on  overall  support  for 
technology  in  the  school,  two  questions  addressed  the  parent,  community  and  administrative  support  necessary 
for  technology  to  impact  teaching  and  learning,  both  with  scores  of  5 = “Strongly  Agree.”  Within  the  technical 
support  section,  the  teachers  concurred  that  they  could  readily  answer  technology  related  questions  with  a score 
of  4 = “Agree.”  Within  the  section  on  attitudes  toward  the  laptop  program,  two  questions  represented  the 
teachers’  enthusiasm  for  the  program  and  confidence  about  their  abilities,  both  with  scores  of  5 = Strongly 
Agree.  Also  notable  is  a question,  “School  computers  are  well  maintained,”  which  received  the  lowest  mean 
score  of  3.75  between  “Neither  Disagree  nor  Agree”  (3)  and  “Agree”  (4).  This  question  while  receiving  the 
lowest  mean  score  also  had  the  largest  amount  of  variance  among  the  respondents  ( SD  = 1.26). 

Technology  Skills  Assessment  (TSA).  The  TSA  revealed  very  high  levels  of  confidence  by  the  fifth 
grade  teachers  (N= 4)  to  use  technology  throughout  six  basic  areas:  computers,  software,  multimedia,  Internet, 
advanced  skills,  and  using  technology  for  learning.  Teacher  confidence  was  high  in  all  six  areas  with  mean 
scores  of  2.5  or  higher,  between  “Somewhat”  and  “Very  Easily”  and  very  little  discrepancy  among  their  ratings 
(SD=. 05  to  .38).  The  teachers  rated  themselves  highest  in  Computer  Basics  (47=2.98)  and  Software  Basics 
(M= 2.96).  Remarkable  is  that  of  the  47  questions  on  the  TSA,  the  teachers  rated  their  confidence  in  30  of  the 
tasks  very  easily,  which  constitutes  63.8%  of  the  tasks.  Moreover,  of  the  47  questions,  only  4 questions  were 
rated  below  2 (“Somewhat”). 
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Focus  Group  Results 

Two  focus  group  interviews  each  were  conducted  with  the  four  fifth  grade  teachers  and  approximately 
8 to  10  of  the  fifth  graders.  Initial  interviews  were  conducted  in  Fall  2003  and  follownip  interviews  in  May 
2004.  Interview  questions  centered  on  three  topic  areas:  (a)  uses  of  technology,  (b)  expectations  of  the  initiative 
and  (c)  reservations  with  the  initiative.  Verbatim  comments  are  enclosed  in  quotation  marks  to  represent  most 
accurately  the  voice  of  the  students  and  teachers. 

Teacher  Focus  Group.  The  teachers  primarily  discussed  two  themes:  (a)  computer  use  and  (b) 
pressures  and  concerns. 

Computer  use.  The  teachers  identified  online  tests,  publishing  stories,  information  seeking  and 
research  on  the  Internet,  word  processing,  electronic  presentations,  and  draw/paint  applications  as  the  computer 
uses  they  had  implemented.  They  also  considered  it  their  responsibility  to  teach  the  students  about 
trustworthiness  with  the  laptops  and  the  “upkeep”  for  the  laptops.  The  teachers  were  proud  of  the  very  few 
numbers  of  computers  that  had  been  “dropped”  by  the  students.  They  also  felt  in  order  to  leverage  the  use  of 
the  laptops,  it  was  necessary  to  teach  “saving,”  “how  to  save”  and  management  routines  for  using  the  laptops. 

The  teachers  described  that  they  would  like  to  use  the  laptops  for  about  “50%”  of  the  time,  about  “two 
to  three  hours  a day.”  However,  they  felt  some  challenges  prevented  them  from  achieving  this  goal. 

Specifically,  the  scope  of  the  curriculum,  as  well  as  state  and  district  standards  (i.e.,  “student  performance 
indicators”),  was  difficult  to  achieve  with  and  without  using  the  laptops.  Changes  in  the  district  curriculum  also 
made  it  “difficult  to  schedule”  and  plan  ahead  for  the  next  year. 

Pressures  and  concerns.  In  the  fall  semester,  the  teachers  were  concerned  about  the  keyboarding  skills 
of  the  students,  as  well  as  “maintenance,”  “upkeep”  and  “technical  support”  for  using  the  laptops.  However,  in 
the  spring  semester,  the  teachers  voiced  that  they  were  “confident”  in  the  use  of  the  laptops  for  instruction  and 
that  they  could  “figure  out”  most  of  the  technical  problems  or  questions  that  arose.  They  also  noted  that  they 
depended  on  one  another  for  expertise.  Proximity  to  one  another’s  classrooms  facilitated  this  learning 
community. 

Similarly,  in  the  fall  semester,  they  described  that  their  colleagues  were  “jealous”  toward  the  fact  that 
they  have  laptops;  however,  their  colleagues  were  relieved  they  “did  not  have  to  deal  with  the  responsibilities 
and/or  tasks”  that  accompany  using  the  laptops  for  learning.  In  the  spring,  this  perception  continued.  One 
teacher  described  it  as  “laptop  envy.”  They  felt  that  some  teachers  perceived  the  laptop  program  as  an  “extra 
burden.”  Indicative  of  their  growth  and  comfort  levels,  however,  the  teachers  noted  that  colleagues  “come  to  us 
for  questions”  about  using  computers  or  technical  problems,  demonstrating  pride  in  their  accomplishments. 

In  the  fall  semester,  the  teachers  expressed  “meeting  expectations  of  the  administration”  as  pressure 
toward  using  the  laptops,  and  “meeting  standards”  defined  by  the  school  district  and  state  as  pressures  they  felt 
toward  not  using  the  laptops.  It  is  simple  to  discern  the  conflict  these  pressures  presented  toward  use  and  non- 
use. However,  in  the  spring  semester  they  seemed  to  have  reconciled  these.  The  teachers  concurred  that  they 
did  not  feel  “as  much  pressure  as  in  the  beginning.”  Some  pressure  may  have  been  intrinsic — applied  by  the 
teachers — in  addition  to  the  extrinsic  pressures  they  identified.  Again,  pride  in  their  endeavors  seemed  to  have 
mitigated  the  former  pressures. 

In  the  fall  semester,  concerns  about  the  laptop  program  centered  on  covering  content  and  teaching 
skills  and  knowledge  that  specifically  may  not  be  revisited  until  later  grades.  The  teachers  were  also  concerned 
that  they  would  be  teaching  computer  skills  that  the  students  “will  not  use  in  other  grade  levels.”  In  the  spring, 
concerns  shifted  to  improvements  for  the  following  school  year.  For  example,  another  teacher  would  be  added 
to  the  fifth  grade.  Currently,  the  core  subject  areas  (language  arts,  mathematics,  science  and  social  studies) 
were  taught  by  one  of  each  of  the  fifth  grade  teachers.  So,  all  the  fifth  grade  students  rotated  through  each  of 
the  teachers.  In  fall  2004,  another  teacher  will  be  added  and  she  will  most  likely  share  teaching  responsibilities 
for  one  or  more  content  areas.  Integrating  this  teacher  into  the  learning  community  and  “logistically  figuring  it 
out”  how  to  use  the  laptops  when  the  same  subject  matter  is  taught  in  two  different  classrooms  were  indicated 
as  challenges  and  concerns.  In  addition,  technical  difficulties,  such  as  reliability  with  internal  network,  external 
Internet  and  server  access,  continue  to  be  concerns  for  the  teachers.  They  do  feel,  however,  that  they  are  more 
sensitive  to  these  challenges  given  their  reliance  on  these  resources. 

Student  Focus  Group.  Students’  discussions  centered  on  two  themes:  (a)  computer  use  and  (b)  students  that 
benefit  from  using  laptops. 

Computer  use.  The  students  identified  electronic  presentations,  writing  “stories,”  graphing  such  as  an 
ordered  pairs  lesson  in  mathematics,  draw/paint  to  create  an  original  flower  in  science  and  visiting  Internet  sites 
for  information  seeking/research  as  ways  they  had  used  the  laptop  computers  in  class.  They  said  in  many 
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instances  using  the  laptop  computers  was  “easier,”  “fun  and  faster”  and  required  less  “writing.”  But  they  also 
recognized  that  in  some  in  cases,  such  as  with  the  flower  project  in  science,  it  was  “harder”  than  completing  the 
assignment  on  paper,  where  they  previously  used  craft  materials. 

The  students  reported  issues  that  they  felt  prevented  from  using  them  more  in  class.  These  related  to 
technical  and  logistical  topics  as  well  as  personal  responsibility.  Technical  problems,  such  as  system  crashes, 
“freezing,”  glitches  in  the  application  programs,  system  updates,  as  well  as  broken  keypads  and  broken  cords 
requiring  “repair  time,”  continued  to  be  challenges  throughout  the  school  year.  Logistical  issues  primarily 
centered  on  lack  of  “battery  power,”  which  sometimes  caused  delays  in  instruction  and  in  a few  instances 
resulted  in  lost  work. 

A very  strong  theme  for  the  students’  use  was  personal  responsibility  for  the  laptops.  This  message  as 
described  by  the  teachers  in  their  interviews  appeared  to  have  been  respected  by  the  students.  Many  students 
felt  that  they  “had  to  be  careful  when  using  the  laptops”  so  they  would  not  damage  them.  They  also  felt  that 
“not  being  responsible”  would  prevent  them  from  being  able  to  use  them.  Likewise,  “dropping  the  laptops”  was 
also  a concern  for  the  students.  But,  they  also  felt  like  they  were  “gaining  responsibility”  with  using  the  laptops 
and  taking  care  of  them. 

At  the  beginning  of  the  school  year,  students  thought  typing,  or  keyboarding,  skills  were  a problem  for 
students,  preventing  use  or  slowing  use.  They  even  suggested  requiring  “a  typing  class.”  However,  at  the  end 
of  the  year,  the  students  did  not  include  this  with  their  challenges  to  using  the  laptops  or  with  the  students  they 
felt  benefited  most  from  the  laptops’  use. 

It  is  also  interesting  to  note  that  a few  students  also  felt  that  they  “can’t  work  on  [the  laptops]  all  day.” 
They  felt  it  was  implausible.  Similarly,  students  thought  if  they  used  the  laptops  “all  day,”  it  “might  get 
boring.” 

Students  that  benefit  from  using  laptops.  The  fifth  graders  felt  that  the  types  of  students  who  benefited 
most  from  using  the  laptops  were  those  that  were  fast  learners  or  “students  who  learn  more.”  They  also  thought 
those  peers  that  had  prior  knowledge  about  computers  or  were  '"into  computers ” also  benefited.  In  the  fall  they 
felt  that  students  who  had  “high  IQs”  benefited,  but  in  the  spring,  students  mentioned  that  “students  who  are  not 
as  smart”  were  the  ones  benefiting  most. 

The  students  agreed  in  the  fall  and  spring  that  the  type  of  students  who  did  not  benefit  from  using  the 
laptops  were  those  students  who  “don’t  care”  and  those  who  “don’t  listen”  or  do  not  pay  attention.  One  student 
called  these  indifferent  and  lackadaisical  students  “goofers,”  explaining  that  they  “sit  there  and  do  nothing.” 
They  also  felt  that  the  “smartest  kids  in  class”  benefited  least,  because  “they  already  know”  how  to  use  the 
laptops  and  are  confident  with  the  content.  Similarly,  the  “fast  learners”  they  felt  should  have  additional 
resources,  because  they  are  “held  back”  when  skills  are  retaught. 

Discussion 

The  discussion  of  the  findings  of  this  study  is  presented  in  association  with  each  of  the  major  research 
questions  in  the  respective  sections  below.  The  limitations  associated  with  this  study  are  relative  to  all 
qualitative  research.  The  small  sample  limits  the  ability  to  generalize  these  finding  to  larger  populations.  More 
specifically,  this  research  represents  the  voices  of  students  and  teachers  in  a suburban  city,  so  it  is  impossible  to 
say  if  these  findings  would  extrapolate  to  other  populations.  As  such,  these  results  should  be  interpreted  with 
caution,  and  the  extent  to  which  these  results  can  be  applied  in  other  contexts  is  situated  with  the  reader. 

In  what  ways  has  the  effectiveness  of  instruction  through  the  use  of  student  laptop  computer  been 
impacted? 

While  it  is  difficult  to  determine  increases  without  baseline,  or  beginning,  data,  SOM  results  indicate 
extensive  uses  of  cooperative/collaborative  learning,  project-based  learning  and  the  teachers  acting  as  coaches 
or  facilitators.  Results  from  the  SCU  indicate  extensive  uses  of  productivity  tools,  specifically 
draw/paint/graphics  and  electronic  presentations,  and  Internet  research  with  Internet  browsers.  SCU  results  also 
suggest  wide  use  across  the  content  areas.  Finally,  the  overall  meaningfulness  of  the  computers  was  observed 
to  be  extensive  in  approximately  one-third  to  fifty  percent  of  the  classroom  visits.  The  results  from  the  SOM 
coupled  with  results  from  the  SCU  point  to  activities  that  result  in  meaningful  uses  of  computers  that  were 
based  on  problems,  required  critical  thinking  skills  and  used  computer  applications  to  locate,  process  and/or 
manipulate  information.  Despite  the  limited  scope  of  technology  tools,  those  tools  observed  were  seen  to 
meaningfully  integrate  technology  to  enhance  student  learning. 

Moreover,  data  from  the  RSCA  revealed  that  teachers  used  technology  with  over  40%  of  their  student- 
centered  learning  activities.  This  finding  is  not  surprising  given  that  the  observations  were  conducted  with 
targeted  lessons,  where  the  teachers  were  asked  to  demonstrate  technology  integration.  However,  specifically 
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notable  is  that  technology  was  observed  in  every  instance  with  project-based  learning.  This  is  a significant 
accomplishment  given  the  focus  of  the  initiative  on  using  technology  to  impact  teaching  and  learning.  Plus, 
technology  for  student  use  is  often  employed  best  during  more  ill -defined  learning  contexts  such  as  project-  and 
problem-based  learning  (Morrison  & Lowther,  2005). 

It  is  important  to  indicate  that  during  two  of  the  observations  as  noted  in  the  SCU,  the  school  network 
access  was  intermittent.  As  the  laptop  program  continues  and  potentially  expands,  school  network  reliability 
will  affect  the  effectiveness  of  instruction  with  the  increased  dependence  on  digital  resources,  such  as  the 
Internet,  email  and  school  servers.  Other  implementations  (e.g., Edwards,  2003)  have  als  o experienced 
challenges  with  unstable  or  unreliable  networks.  While  statistics  may  demonstrate  promise  for  improving 
access  to  technology  and  digital  resources  (National  Center  for  Educational  Statistics,  2001),  the  reliability  of 
school  networks  may  imp  ede  or  dissuade  teachers  from  developing  and  implementing  lessons  that  require 
Internet  and  intranet  access. 

To  what  degree  and  in  what  ways  have  teachers  integrated  technology  with  classroom  instruction? 

The  proposed  laptop  program  included  comprehensive  technology  integration  training  for  the  fifth 
grade  teachers;  this  was  not  implemented  due  to  cost  and  lack  of  grants  support.  While  no  extensive  training 
was  conducted,  the  fifth  grade  teachers  participated  in  a workshop  on  using  Microsoft  PowerPoint  in  Fall  2003 
and  informal  discussions  about  classroom  management  with  laptops  during  Spring  2003.  Moreover,  the 
teachers  through  focus  group  interviews  indicated  they  had  participated  in  professional  development  workshops 
offered  through  the  local  school  district.  They  also  relied  heavily  on  one  another  to  extend  their  expertise, 
creating  an  informal  community  of  practice  (Wenger,  1998)  leveraged  from  their  grade  team.  So,  primarily  the 
teachers  used  their  educational  philosophies  and  pedagogy  to  envision  effective  technology  integration.  Pierson 
(2001)  suggests  that  pedagogical  expertise  and  teacher  epistemologies  influence  technology  integration. 
Likewise,  teachers’  personal  technology  skills  impact  the  meaningfulness  of  the  technology  integration 
activities  as  well  as  the  instruction  and  assessment.  While  this  evaluation  did  not  seek  to  explore  teacher 
epistemologies  or  pedagogical  expertise,  there  is  some  evidence  from  observations  and  focus  group  interviews 
to  suggest  the  fifth-grade  teachers’  visions  for  technology-enhanced  teaching  and  learning  represents  the 
intersection  of  exemplary  technological  ability  and  exemplary  teaching  ability,  Pierson’s  Category  4. 

In  the  fall  semester,  both  teachers  and  students  mentioned  keyboarding  skills  as  impediments  to 
technology -enhanced  teaching  and  learning.  As  mentioned  earlier,  the  students  actually  suggested  requiring  “a 
typing  class.”  However,  in  the  spring,  keyboarding  skills  were  not  included  as  an  obstacle  to  using  the  laptops 
computers  more.  SCU  results  reported  keyboarding  skills  were  very  good  in  all  classroom  visits  (100%).  So 
student  keyboarding  skills  seemed  to  have  improved  throughout  the  year  and  may  be  part  of  the  learning  curve 
for  laptop  initiatives. 

To  what  degree  do  laptop  teachers  use  methodologies  that  stress  higher-order  learning  and  student- 
centered  learning  activities? 

In  almost  90%  (88.9%)  of  the  targeted  classroom  visits,  teachers  were  observed  extensively  to  be 
acting  as  a coach  or  facilitator  of  learning.  Other  activities  indicative  of  critical  thinking  and  student 
engagement  were  seen  in  over  30%  of  the  visits.  Cooperative/collaborative  learning,  which  was  observed 
extensively  in  33.3%  of  the  visits,  was  observed  to  be  a somewhat  strong  or  strong  application  in  over  40%  of 
the  observations  (44.4%).  Project-based  learning,  which  was  observed  in  100%  of  the  visits,  was  observed  to 
be  somewhat  strong  or  strong  application  in  over  75%  of  the  visits  (77.7%).  Finally,  independent 
inquiry/research,  which  was  observed  in  over  40%  of  the  visits  (44.4%),  was  observed  to  be  a somewhat  strong 
or  strong  application  in  approximately  10%  of  the  visits  (11.1%).  These  data  indicated  some  use  of  non- 
traditional,  or  more  student-centered,  instructional  methods.  Similar  initiatives  (e.g.,  Bickford  et  ah,  2002)  have 
reported  challenges  with  teachers’  uses  of  student -centered  pedagogy.  Pierson  (2001)  agreed  that 
characterizations  of  exemplary  technology-using  teachers  represent  a combination  of  content  knowledge, 
pedagogical  knowledge  and  technological  knowledge  that  few  teachers  may  achieve. 

The  results  from  this  research  suggest  the  fifth-grade  teachers  have  the  pedagogical  knowledge  and 
skill  to  implement  teaching  methods  that  emphasize  higher-order  and  critical  thinking.  While  the  observations 
used  in  this  study  were  pre-arranged  visits,  it  is  impossible  to  determine  the  regularity  of  these  strategies 
throughout  the  school  year.  In  addition,  with  a lack  of  baseline  comparative  data,  it  is  also  impossible  to  discern 
whether  the  fifth-grade  teachers  employed  these  methods  prior  to  the  laptop  initiative.  However,  professional 
development  efforts  or  program  goals  focused  more  on  these  student -centered  strategies  may  increase  the 
frequency  with  which  they  are  used. 

To  what  degree  has  the  laptop  program  impacted  teacher  attitudes  toward  technology? 

The  fifth  grade  teachers  were  enthusiastic  about  the  laptop  program.  Succinctly,  the  teachers  felt  the 
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program  had  positively  impacted  their  classroom  instruction  and  positively  impacted  the  fifth  grade  students. 
Moreover,  the  teachers  felt  they  were  ready  to  integrate  technology  into  their  instruction.  This  was  corroborated 
by  the  focus  group  interviews  as  the  teachers  discussed  their  “confidence’'  with  the  laptops  and  reconciliation 
with  previous  intrinsic  and  extrinsic  pressures.  The  TSA  highlighted  the  teachers’  expertise  in  computer  basics, 
software  basics,  multimedia  basics,  Internet  basics,  advanced  skills  and  using  technology  to  support  learning. 
This  corroborates  previous  research  (e.g.,  Pierson,  2001;  Silvernail  & Lane,  2004)  that  suggests  teacher 
technology  skills  positively  impact  technology  integration. 

Personal  professional  development  objectives  or  grade  level  professional  development  goals  should 
consider  the  few  technology  skills,  such  as  Boolean  strategy  searches,  electronic  communications  other  than 
email  and  electronic  teaching  portfolios,  that  received  mean  scores  below  2 (between  “Not  at  All”  and 
“Somewhat”).  These  skills  should  be  evaluated  for  their  value  to  the  teachers,  students  and  relevance  to  the 
program  as  the  initiative  continues. 

Finally,  the  teachers  felt  they  have  the  support  of  parents,  the  community,  the  administration  and  the 
technical  support  necessary  to  be  effective  with  technology  integration  and  improve  student  learning.  Silvernail 
and  Lane  (2004)  reported  that  success  with  technology  integration  also  appeared  to  be  influenced  by  key 
individuals  to  champion  the  program.  The  parents  and  school  administration,  while  initially  imposing  extrinsic 
pressure  on  the  teachers,  may  in  fact  have  translated  into  the  types  of  advocates  necessary  to  support  the  laptop 
program. 


Conclusion 

From  the  formative  results  in  this  study,  the  school  had  positive  teacher  technology  competence  and 
confidence,  used  instructional  strategies  that  centered  on  and  facilitated  student  learning  and  employed 
classroom  practices  that  engaged  students  in  meaningful  technology-supported  activities.  Pierson’s  (2001)  case 
studies  emphasized  content,  pedagogical  and  technological  knowledge  as  factors  influencing  technology 
integration.  Windschitl  and  Sahl  (2002)  illuminated  further  this  work  by  contending  that  teacher  beliefs 
coupled  with  school  culture  and  perceived  support  impact  technology  integration,  specifically  in  laptop 
programs.  In  the  present  study,  the  fifth-grade  teachers’  progress  toward  more  student -centered  activities, 
including  project-based  learning,  cooperative/  collaborative  learning  and  acting  as  a coach  or  facilitator,  seems 
to  support  this.  The  interdependence  with  one  another  and  community  of  practice  established  by  the  fifth  grade 
teachers  also  appears  to  have  created  an  informal  culture  of  support  and  knowledge  sharing  (Wenger,  1998). 
While  the  teachers  may  have  been  novices  at  the  beginning  of  the  school  year,  clearly  all  the  teachers  felt  they 
were  skilled  enough  to  use  a variety  of  software  applications  for  meaningful  learning.  These  three  factors  — 
teacher  pedagogical  knowledge,  technological  knowledge  and  a supportive  culture  — seem  to  be  strong 
indicators  for  impacting  technology  integration  at  this  school. 
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Using  Human  Performance  Technology  (HPT)  to  Identify  Potential 
Barriers  to  Online  High  School  Course  Development 
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Gloria  Llama 
Jennifer  Mansfield 
Florence  Martin 
James  Klein 

Arizona  State  University 

Executive  Summary 

A suburban  high  school  district  in  Arizona  was  recently  named  as  a provider  of  Technology  Assisted 
Project  Based  Instruction  (TAPBI).  The  district’s  challenge  was  to  create  20  effective,  online  high  school 
courses  in  approximately  nine  months.  Successful  deployment  depended  in  large  measure  on  the  ability  of  the 
teachers  of  those  courses  to  effectively  design  and  develop  the  instructional  content.  Therefore,  the  district  had 
to  train,  prepare,  and  support  the  teachers  in  their  work. 

To  help  identify  factors  that  might  prevent  teachers  from  meeting  the  challenge,  a team  of  students 
from  Arizona  State  University  worked  closely  with  the  district  during  a two -month  period  as  the  TAPBI 
program  began.  This  paper  documents  the  data  collection  methods  employed,  findings,  and  subsequent 
recommendations  for  attaining  success. 


Project  Purpose 

The  purpose  of  the  project  was  to  assist  the  district  in  identifying  potential  barriers  to  effective  design 
and  development  of  20  online  high  school  courses.  The  district  had  been  named  a provider  of  Technology 
Assisted  Project  Based  Instruction  (TAPBI),  which  required  it  to  provide  a “variety  of  educational 
methodologies  employed  by  the  school  and  the  means  of  addressing  the  unique  needs  and  learning  styles  of 
targeted  pupil  populations  including  computer  assisted  learning  systems,  virtual  classrooms,  virtual  laboratories, 
electronic  field  trips,  electronic  mail,  virtual  tutoring,  on-line  help  desk,  group  chat  sessions  and  non-computer 
based  activities  performed  under  the  direction  of  a certificated  teacher.”  The  district  wanted  to  identify 
strategies  leading  to  optimal  implementation  of  the  program. 

Data  Collection  Methods 

Data  collection  for  the  project  employed  four  methods:  analysis  of  a survey  administered  by  the  district 
to  teachers,  follow-up  interviews  with  teachers  who  were  interested  in  the  project  and  who  had  completed  the 
survey,  observation  of  an  informational  meeting  for  interested  teachers,  and  a review  of  relevant  literature. 

Survey 

The  district  developed  a survey  instrument  to  be  administered  online  to  the  teachers  throughout  the 
district.  The  instrument  introduced  the  TAPBI  program  to  the  teachers  and  asked  for  responses  from  those  who 
had  an  interest  in  participating  as  designers  and  developers  of  online  courses.  The  items  used  for  data  collection 
focused  primarily  on  the  teachers’  skill  and  experience  in  online  environments.  The  district  provided  the  project 
team  with  the  data  obtained  from  survey  responses. 

Interviews 

Based  on  the  survey  responses,  the  project  team  divided  teachers  into  three  categories  according  to 
their  skill  and  experience  designing  and  delivering  instruction  in  online  environments.  The  project  team  then 
contacted  each  teacher  via  email  in  an  effort  to  arrange  a follow-up  interview.  The  purpose  of  the  interview  was 
to  understand  teachers’  perceptions  of  the  TAPBI  program,  and  to  gauge  their  expectations  regarding  potential 
barriers  to  effective  implementation.  Select  administrators  were  also  contacted  for  similar  reasons. 

Informational  Meeting 

The  project  team  attended  an  informational  meeting  held  for  teachers  interested  in  participating  in  the 
program.  Team  members  who  attended  the  meeting  were  able  to  listen  to  teachers’  questions  and  identify  some 
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of  their  concerns  regarding  the  program. 

Literature  Review 

We  anticipated  that  our  task  of  identifying  potential  barriers  to  effective  design  and  development  could 
be  aided  by  drawing  upon  the  experiences  of  other  organizations  that  had  implemented  similar  programs  and 
had  published  their  experience  in  the  literature.  Therefore,  we  searched  through  relevant  journals  for  cases  in 
which  organizations  had  implemented  a program  similar  to  TAPBI,  and  identified  barriers  to  effective 
implementation. 


Findings 

Analysis  of  the  data  led  to  several  findings  with  potential  implications  for  action  on  the  part  of  the 
district.  Based  on  these  findings,  we  generated  recommendations  for  how  to  prevent  performance  barriers  to  the 
TAPBI  program. 

There  are  a wide  variety  of  potential  barriers  to  an  effective  rollout  of  the  program.  These  barriers  can 
be  categorized  into  three  general  groups:  Administrative  / Strategic;  Experience  / Knowledge;  and  Motivational 
/ Incentive.  Our  discussions  with  individual  teachers  revealed  that  they  had  several  fears  about  their 
participation  in  the  program,  and  that  many  lack  experience  as  online  teachers  and  designers.  Our 
communication  with  administrators  brought  to  light  that  the  district  is  already  grappling  with  issues  such  as 
marketing,  teacher  contracts,  and  copyright.  The  long-term  success  of  the  program  rests  on  effectively  dealing 
with  many  of  these  issues. 

Teachers  and  administrators  view  lack  of  time  as  a potential  barrier.  Devising  strategies  to  lessen 
encroachment  of  program  responsibilities  on  the  teachers’  pre-existing  classroom  responsibilities  should  be  a 
priority. 

Many  of  the  teachers  lack  critical  experience  and  knowledge  about  designing  online  instructional 
content.  This  relates  to  using  WebCT  and  supporting  multimedia  technology.  The  gap  that  exists  between  many 
teachers’  knowledge  of  the  technology  necessary  for  development  and  the  knowledge  needed  to  effectively 
develop  their  courses  must  be  bridged  for  the  program  to  be  successful.  However,  we  felt  that  the  teachers  had  a 
good  understanding  of  how  their  online  courses  would  differ,  pedagogically,  from  their  classroom  courses. 

Teachers  expect  and  welcome  peer  review  of  their  course  designs.  This  strategy  that  the  district  is 
planning  to  implement  can  be  a very  effective  method  of  ensuring  sound  pedagogical  design,  and 
standardization  of  technology,  content,  and  designs.  Enthusiasm  for  peer  review,  though,  could  become  fragile, 
and  could  easily  shatter  if  the  reviews  are  not  done  constructively  and  meaningfully. 

Implications  and  Recommendations 

The  implications  for  an  optimal  rollout  of  the  TAPBI  program  depend  on  three  key  factors:  time, 
training,  and  support. 

Time 

This  is  the  most  crucial  factor  for  developing  the  TAPBI  program.  Although  most  teachers  said  they 
could  complete  their  course  by  the  deadline,  time  was  the  top  concern  for  both  teachers  and  administrators.  At 
the  time  of  this  report,  the  courses  to  be  developed  had  just  been  selected,  but  not  yet  approved  for  funding  by 
the  District  Governing  Board. 

Training 

The  data  clearly  demonstrate  that  the  teachers  are  experts  in  their  content  area,  but  they  want 
multimedia/online  course  training.  This  is  supported  by  the  initial  survey  in  which  over  half  of  the  respondents 
indicated  they  had  not  developed  course  components  for  students  using  the  online  WebCT  environment.  Most 
feel  that  they  need  some  kind  of  multimedia/online  course  training  so  they  can  develop  classes  that  are  going  to 
meet  the  needs  of  their  virtual  students.  Literature  reviews  also  recognized  that  courses  with  text -heavy  content 
are  insufficient  to  retain  student  interest,  and  can  lead  to  high  attrition  rates. 

Support 

The  teachers  must  feel  that  this  project  is  going  to  be  supported  by  the  district  with  IT  resources, 
funding,  and  management.  In  addition,  some  instructors  have  been  involved  with  similar  programs  that  have 
never  been  brought  to  fruition,  and  they  fear  a lack  of  support  could  undermine  their  work  developing  the  online 
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courses  for  the  TAPBI  program. 

The  following  interventions  are  recommended  for  the  TAPBI  program  rollout  effort  and  to  minimize 
potential  barriers  for  optimal  success. 

Vision  and  Strategies  Structure 

Document  the  vision  and  strategy  of  the  TAPBI  program  as  it  applies  to  the  district  to  instill 
confidence  in  the  teachers.  This  should  include  buy-in  from  the  District  Governing  Board  and  pledged  IT 
support  for  the  program.  A statement  documenting  the  District’s  vision  and  strategies  should  be  included  in  the 
development  packet  given  to  the  teachers  when  they  are  selected  to  develop  a course  as  proof  of  commitment  to 
the  program. 

Multi-year  Strategic  Plan 

Develop  a schedule  to  keep  course  development  on  track,  including  a timetable  of  deadlines  for  each 
step  in  the  process.  This  lets  the  teachers  know  what  they  are  expected  to  have  completed  by  when,  and  alerts 
the  district  to  those  teachers  who  may  be  having  difficulties  and  are  falling  behind.  This  strategic  plan  also 
allows  the  district  to  see  a problem  as  it  arises  and  intervene  in  time  to  correct  it. 

Mentoring  Program/Peer  Coaching 

Set  up  development  groups  with  an  experienced  teacher  as  a mentor  to  help  those  with  less  experience. 
Provide  time  for  them  to  meet  in  person  to  demonstrate  “hands-on”  procedures.  Provide  a Discussion  Board  for 
peer  coaching  and  communication  with  administrators.  Mentors  should  provide  examples  of  both  effective  and 
less-effective  online  course  designs  as  examples  to  the  less-experienced  teachers. 

WebCT  and/or  Multimedia  Training 

Arrange  for  expert  trainers  to  come  to  campus  for  “hands-on”  training.  Select  one  individual  from  each 
campus  as  a liaison  between  the  teachers  and  trainers  to  stay  current  with  software  programs  and  techniques. 

The  liaison  will  then  share  information  at  mentor  meetings  or  through  the  Discussion  Board.  Again,  training 
should  provide  both  effective  and  less-effective  examples  of  online  design  for  the  teachers  to  reference. 

Feedback/Appraisal  System 

Schedule  course  material  reviews  at  logical  intervals  (steps)  throughout  the  development  process  as 
shown  in  the  Strategic  Plan.  This  should  be  an  ongoing  formative  evaluation.  By  waiting  until  course 
completion  for  the  jury  review,  it  may  be  too  late  to  meet  the  required  deadline  if  course  correction  is  required. 
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Table  1 Results  of  Interviews  with  Teachers  and  Administrators 


1.  What  do  you  see  as  likely  problems  for  you  in  the  development  of  your  online  courses?  (Arrange  in  the  order 
of  importance  with  1 being  the  most  important  and  7 being  the  least  important.) 


Teacher 

Administrator 

Mean 

Mean 

Lack  of  time 

1.50 

1.00 

Lack  of  multimedia  skills 

3.50 

4.00 

Lack  of  District  support 

3.50 

5.00 

Lack  of  IT  support. 

3.67 

5.00 

Lack  of  instructional  design  skills 

3.83 

2.50 

Lack  of  motivation  after  start  of  project 

5.17 

3.50 

Lack  of  content  expertise 

6.83 

7.00 

2.  What  do  you  feel  is  important  to  know  before  you  begin  creating  your  online  course?  (Arrange  in  the  order  of 
importance  with  1 being  the  most  important  and  6 being  the  least  important.) 


Teacher 

Administrator 

Mean 

Mean 

Experience  teaching  a course  online 

2.83 

4.50 

Subject  matter 

3.00 

1.00 

Instructional  design 

3.33 

2.50 

Multimedia  programs 

3.50 

4.50 

Experience  taking  a course  online 

4.17 

3.00 

More  about  computer  technology 

4.67 

5.50 

3.  What  are  your  biggest  fears  in  developing  an  online  course?  (Arrange  in  order  with  1 being  the  biggest  fear 
and  6 being  the  smallest  or  least  fear.) 


Teacher 

Administrator 

Mean 

Mean 

Time 

1.67 

2.00 

Multimedia  skills 

2.50 

3.00 

IT  support 

3.67 

4.00 

Computer  skills 

4.00 

2.00 

Submitting  the  course  to  the  jury 

4.17 

4.50 

Beta-testing  the  course 

5.00 

5.00 

4.  What  support/help  will  the  District  need  to  provide  as  you  design  and  develop  your  course? 

# of  Teacher  Responses 


Training/Workshop  on  WebCT  3 

Real-time  Technology  Support  3 

Training/Equipment  on  Multimedia  Content  1 

Documentation/Training  Manual  for  WebCT  (List  of  tips)  1 

Reviewer  of  Course  Material  1 
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5.  How  will  your  online  course  be  designed  differently  than  your  classroom  course? 


# of  Teacher  Responses 


Online  Quizzes  1 

Multimedia  Items  (Audio  and  Visual)  2 

Discussion  Forums  1 

Online  Resources  (Materials)  2 

Student  Centered  Format  1 

Independent  Research  2 

Critical  Thinking  1 

Reworking  the  content  for  assessment  and  motivational  factors  1 

Drill  and  Practice  and  Feedback  on  Quizzes  1 


6.  What  sort  of  evaluation  procedures  will  you  be  following  (if  any)  during  the  design  and  development  of  your 
course? 


Evaluation  by  Peers  3 

Field  Test  (Beta  test)  3 


7.  Are  you  aware  of  any  standards  that  need  to  be  met  in  the  design  and  development  of  these  courses? 


AIMS  Standards  1 

ISTE  Standards  1 

Local,  State  and  National  Standards  4 

Self  developed  standards  from  online  chats  1 

Standards  given  by  Project  Coordinator  1 

Good  Instructional  Design  Principles  1 

IT  Standards  1 


8.  What  resources  has  the  school  has  given  you  for  this  project? 


Textbooks  1 

Question  banks  1 

Teacher  editions  1 

Computer  resources  2 

Active  Discussion  Board  1 

WebCT  LMS  and  Tutorial  Access  1 

9.  What  resources  would  help  you? 

Training  on  WebCT  2 

Training  on  IT  (Integration  of  Technology)  1 

Internet  Access  1 

Enough  computers  to  beta  test  1 

CD  Burner  1 

Making  Video  Excerpts  Available  Online  1 

10.  What  was  your  primary  motivation  in  getting  involved  in  this  project? 

Curiosity  1 

Use  of  technology  2 

Wish  to  become  an  online  teacher  later  2 

Money  1 

Helps  Gifted  Children  and  in  home  schooling  1 
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1 1 .  What  do  you  perceive  as  being  the  biggest  obstacle  for  this  project? 


Time  involved  in  uploading  the  materials  6 

Lack  of  knowledge  of  WebCT  4 

Funding  problems  1 

Tech  Support  2 

Loosing  face  to  face  contact  1 

Security  issues  online  (students  could  cheat)  1 

Training  2 

Resources  1 

Lack  of  Coordination  1 

Skills  1 

12.  Do  you  think  that  converting  your  classroom-based  course  to  an  online  course  will  have  the  same  effects  on 
learning? 

Depends  on  the  student  (Discipline,  Motivation,  personality  types)  3 

Online  courses  are  positive  move  forward  in  the  educational  system  1 

Depends  on  the  Challenge  to  overcome  the  lack  of  face-to-face  contact  1 

More  successful  for  motivated  students  1 

13.  Would  you  be  comfortable  teaching  an  online  course  developed  by  someone  else  or  would  you  let  someone 
else  teach  the  course  you  developed? 

Will  let  teach  others  courses  1 

Prefer  to  teach  my  own  course  3 

Willing  to  let  someone  else  teach  my  course  2 

Will  redesign  and  teach  somebody  else  course  3 

14.  If  you  have  developed  online  course  before,  what  do  you  know  now  that  you  wished  you  could  have  known 
then? 

Make  lesson  very  explicit  and  complete  1 

Provide  feedback  and  positive  reinforcement  1 

More  contact  with  students  in  an  online  setting  1 

Be  ready  to  handle  technology  issues  1 

Course  design  takes  more  time  than  anticipated  1 

The  course  design  process  1 

Teaching  and  learning  in  an  online  environment  1 

School/districts  expectations  1 


15.  To  what  extent  do  you  perceive  this  initiative  to  be  of  value  to  you  and  to  the  school  where  you  teach? 

Will  help  the  district  to  lead  in  cutting  edge  opportunities/efforts  1 

School  will  be  in  forefront  of  this  educational  endeavor  1 

Might  end  up  re  -inventing  the  wheel  1 

Good  challenge  as  we  are  moving  into  technology  centered  world  1 

Great  opportunity  for  personal  growth  1 

Good  opportunity  for  students  to  be  successful  1 

Classes  taught  on  one  campus  can  be  available  for  all  the  students  1 
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Which  Way  for  Which  Teacher?  Comparing  Two  Graduate  Courses  that 
Teach  Technology  Integration  to  Teachers 
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V.  J.  McClendon 
Michael  Orey 
Julie  Moore 
University  of  Georgia 

Abstract 

This  brief  paper  describes  a comparison  of  two  different  courses,  both  of  which  have  been  approved  by 
the  state  of  Georgia's  Professional  Standards  Commission  (PSC).  These  courses  fulfill  the  PSC  requirement 
that  all  teachers  be  competent  with  technology  and  know  how  to  integrate  technology  into  teaching  and 
learning  by  the  year  2006.  The  method  of  comparison  was  document  analysis.  Each  of  these  courses  has  been 
approved  to  by  the  state  as  meeting  the  “Special  Technology  Requirement,  ” but  there  are  many  contrasting 
elements.  A description  is  given  of  how  each  course  addresses  the  common  topics  taught  by  the  courses  and 
then  a rating  is  assigned  for  each  course  on  how  thoroughly  it  addresses  each  topic.  After  the  content  analysis 
of  each  class  was  conducted,  data  from  another  study  reporting  the  amount  of  time  spent  by  students  in  the 
courses  studied  was  incorporated  to  add  depth  to  the  comparison  of  the  courses. 

Study  Rationale 

In  2000  the  Georgia  state  legislature  passed  House  Bill  1187  which  became  the  A+  Education  Reform 
Act  of  2000.  A part  of  that  act  requires  that  all  certified  teachers  in  the  state  meet  a technology  requirement  in 
order  to  keep  their  teaching  certificates  current.  There  are  currently  at  least  three  different  ways  that  in-service 
teachers  - teachers  that  have  previous  certification  and  are  currently  teaching  in  Georgia  schools  - can  meet  that 
requirement  at  the  University  of  Georgia.  One  way  is  a professional  development  model  called  InTech  offered 
by  the  University’s  Educational  Technology  Training  Center  (ETTC).  Another  way  is  a class  offered  by  the 
Department  oflnstructional  Technology  called  EDIT  6150  - Introduction  to  Computer-based  Education.  The 
third  option  is  EDIT  6150  offered  as  an  online  class  rather  than  through  a more  traditional  face-to-face  method. 
The  expressed  purpose  of  each  of  these  courses  is  to  prepare  teachers  to  be  able  to  better  use  modern 
technologies  in  their  teaching  practices. 

In  the  literature  there  are  various  examples  of  individual  programs  for  training  teachers  how  to 
integrate  technology  into  their  curriculum.  The  reports  about  these  describe  attributes  of  each  program  and  why 
those  involved  view  these  programs  as  successful.  For  example,  Norton  and  Gonzales  (1998)  describe  a 
regional  educational  technology  agency’s  attempt  to  meet  the  needs  of  the  teachers  in  their  service  area.  They 
point  out  in  their  report  that  there  are  several  components  that  teachers  being  trained  feel  are  key  to  the  success 
of  the  course  and  are  key  to  helping  them  to  reach  the  course  objectives.  Some  of  those  are  the  fact  that  other 
teachers  teach  the  course  and  that  integration  is  emphasized  rather  than  skills.  In  the  same  report,  instructors 
give  input  as  to  what  they  feel  are  significant  components  of  the  course.  Similar  to  the  opinions  of  the  teachers 
taking  the  course,  the  instructors  feel  that  the  philosophy  of  curriculum  integration  of  the  technology  is 
important.  They  also  agree  that  having  a team  of  teachers  teaching  the  course  is  a significant  feature. 

Another  article  discusses  teachers’  levels  of  concern  for  using  technology  (Gonzales,  Pikett,  and 
Ruppert,  2002).  In  this  article,  the  authors  suggest  a relationship  between  a teacher’s  level  of  concern  for  using 
technology  and  their  skills  and  support  to  do  so.  The  more  skills  and  support  that  teachers  have  with  dealing 
with  technology,  the  higher  their  level  of  concern  for  using  technology  in  their  teaching  will  be.  Another 
program  (Moersch,  2001)  takes  the  opposite  approach  and  connects  a teacher’s  current  use  of  technology  to  his 
or  her  need  for  further  professional  development  in  technology.  In  other  words,  rather  than  suggesting  that 
teachers  need  certain  professional  development  to  be  able  to  integrate  technology,  their  current  level  of 
technology  integration  determines  what  kind  of  professional  development  they  might  need. 
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Is  summary,  there  are  examples  of  studies  that  describe  individual  programs  or  courses  and  then  there 
are  studies  that  describe  levels  of  skills  and  experiences  with  technology  of  the  people  taking  the  classes.  It  is 
the  researchers  goal  to  take  information  from  these  studies  and  go  further  in  an  attempt  to  compare  different 
programs  in  an  attempt  to  discern  which  type  of  course  is  best  for  which  type  of  teachers  according  to  their 
nature,  technical  skills  and  past  experiences. 

Research  Problem  and  Questions 

There  are  many  options  for  teachers  who  need  to  meet  the  technology  requirement  that  the  State  of 
Georgia  has  mandated.  A complete  list  of  options  can  be  found  on  the  PSC’s  website  at 

http://www.gapsc.com/ApprovedPrograms/EducationProgram.asp?technology=yes . There  is  a diversity  of  skills 
and  previous  experiences  of  teachers  in  those  classes.  Further,  there  are  many  people  who  are  continuing  to 
develop  new  courses  to  distribute  and  are  modifying  existing  courses  to  meet  this  requirement. 

This  study  seeks  to  compare  two  of  the  existing  courses  that  teach  technology  integration  to  teachers  in 
an  attempt  to  inform  instructors  and  potential  course  participants  of  which  type  of  course  is  best  for  teachers  of 
different  skill  levels  and  experiences.  In  doing  so,  the  researchers  must  ask  these  questions:  What  key  topics  are 
covered  by  each  individual  course?  How  are  different  topics  covered?  Finally,  what  kinds  of  prerequisite  skills 
are  needed  to  be  able  to  be  successful  with  each  topic? 

Research  Design 

Site 

The  site  for  this  study  is  the  Department  of  Instructional  Technology  at  the  University  of  Georgia.  The 
University  of  Georgia  has  one  of  the  largest  Colleges  of  Education  in  the  country,  and  as  such,  serves  not  only  a 
great  many  of  the  state’s  educators,  but  also  educators  that  represent  every  demographic  in  the  state.  An  impetus 
for  this  study  is  to  become  better  informed  of  the  variety  of  options  teachers  have  for  fulfilling  the  state 
technology  recertification  requirements. 

In  order  to  provide  a richer  context  for  the  evaluation  of  student  experience  in  each  course,  time -on- 
task  data  was  also  collected  on  the  courses  using  a web-based  log  tool  (Amiel,  McClendon,  & Orey,  2003). 
Students  were  asked  to  input  the  amount  of  time  spend  every  week  of  class.  The  student  log  categories  represent 
a comprehensive  list  of  time  consuming  activities  that  are  part  of  the  course:  class  time,  group  time  online, 
group  time  face-to-face,  time  spent  working  individually,  technology  problems,  travel,  message  posting/boards, 
and  other.  Data  from  these  logs  were  not  collected  for  comparative  purposes,  but  instead  to  provide  another 
dimension  for  the  analysis  of  in-class  characteristics  (for  a discussion,  see  Clark,  1983,  1994;  Ehrmann,  1995; 
Paulson,  2002). 

Sample  and  Sample  Selection 

This  study  used  as  its  sample  one  section  of  each  of  the  two  courses  during  the  same  semester.  Time  - 
on-task  data  was  collected  on  both  courses.  Participants  were  self-selected  and  received  extra  credit  for 
recording  their  time  -on-task  data  using  the  web-based  log  tool  once  a week.  The  first  course  (online)  had  an 
enrollment  of  21  and  1 1 participants;  the  second  course  (f2f)  had  an  enrollment  of  19  and  10  participants. 

Data  Analysis  and  Procedures 

The  course  materials  including  syllabi,  required  texts,  and  assignment  descriptions  served  as  data  and 
were  analyzed  according  to  the  International  Society  for  Technology  in  Education’s  National  Educational 
Technology  Standards  (ISTE-NETS)  categories.  For  a course  to  be  approved  by  the  PSC  as  meeting  the  special 
technology  requirement,  it  must  address  the  state’s  technology  standards,  which  were  adopted  from  the  ISTE- 
NETS.  In  the  ISTE-NETS  there  are  twenty-three  standards  divided  into  six  diffeient  topical  areas.  A complete, 
detailed  list  of  these  standards  can  be  found  on  ISTE’s  website  at http://cnets.iste.org/teachers/t_stands.html. 
After  it  was  determined  how  each  course  addressed  each  topic,  the  primary  researcher  rated  how  well  that  topic 
was  covered.  She  then  consulted  with  the  other  researchers  to  confirm  her  findings.  In  order  to  rate  how  each 
course  performed  in  addressing  each  topic,  the  researchers  used  a scale  of  one  to  five  with  descriptions  of  what 
would  be  seen  as  evidence  of  covering  the  topic  at  each  level.  The  scale  follows. 

1 - Addressed  only  in  an  introductory  nature  such  as  by  mentioning  in  lecture. 

2 - Addressed  mostly  by  discussion  but  with  some  practice  or  application. 

3 - Addressed  by  an  assignment  that  the  student  must  complete.  Often  covered  as  a secondary 
objective  to  another  assignment. 
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4 - Addressed  by  at  least  one  assignment  and  one  other  method  such  as  discussion,  reading  assignment, 
secondary  objective  of  a lesson,  etc. 

5 - Directly  and  specifically  addressed  by  multiple  assignments  on  the  topic. 

Findings 

Below  is  a description  of  each  course  including  student  demographics  and  information  about  the 
relationship  of  the  researchers  to  the  courses.  The  results  of  the  analysis  are  then  presented  per  ISTE-NETS 
topic  for  each  course. 


Descriptions  of  Each  Course 


EDIT  6150  Face-to-face 

This  course  is  offered  as  a traditional  graduate  level  face-to-face  class  in  15  weekly  meetings  for  3- 
hours  each  over  the  span  of  a 15 -week  semester.  Class  discussions,  technology  skills,  and  software 
demonstrations  are  all  a part  of  the  course.  A wide  variety  of  students  take  this  class  including  in-service  and 
pre -service  teachers,  people  who  work  in  schools  but  not  as  teachers,  and  non-education  majors. 

EDIT  6150  Online 

In  this  online  course,  the  first  class  session  has  a face-to-face  requirement.  All  14  remaining  sessions 
take  place  in  a live  online  classroom.  The  online  environment,  HorizonLive,  includes  slides,  live  demonstrations 
and  2-way  voice -over-IP  audio.  These  live  sessions  last  2 hours  each  and  take  place  over  the  course  of  the  15- 
week  semester.  In  addition  to  the  synchronous  instruction,  a variety  of  asynchronous  requirements  and 
supplements  exist  including  required  discussion  threads,  optional  step-by-step  software  guides,  professionally 
developed  self-instructional  software  (from  NetG),  and  other  materials  and  activities.  The  student  demographics 
of  this  course  are  similar  to  that  of  6150  face-to-face. 

Technology  operations  and  concepts 

Teachers  demonstrate  a sound  understanding  of  technology  operations  and  concepts.  The  face-to- 
face  section  of  EDIT  6150  addresses  technology  operations  and  concepts  through  classroom  activities  and 
several  of  the  course  required  activities  such  as  a word  processing  assignment  and  a PowerPoint  assignment. 
Time  during  class  is  devoted  to  technical  assistance  and  certain  individuals  in  the  class  are  identified  as 
technicians  and  tech  tip  teachers  to  help  those  that  need  personal  assistance. 

Rating:  5 

EDIT  6150  Online  uses  several  different  web-based  and  print-based  resources  to  help  teachers  with 
operations  and  concepts.  Since  the  delivery  of  the  course  is  online,  there  is  no  scheduled  time  that  the  students 
meet  in  person  with  others  in  the  class  or  the  instructor.  However,  the  instructor  does  offer  to  be  available  to  the 
students  if  they  request  a personal  help  session.  The  instructor  also  covers  some  of  the  concepts  during  class 
presentations  delivered  online. 

Rating:  4 

Planning  and  designing  learning  environments  and  experiences 

Teachers  plan  and  design  effective  learning  environments  and  experiences  supported  by  technology. 
Two  of  the  six  deliverable  assignments  for  EDIT  6150  face-to-face  are  lessons  that  are  to  be  designed  for 
implementation  with  students.  Those  deliverables  are  a cognitive  tool  lesson  and  a final  project  that  could  be 
done  in  the  form  of  a WebQuest,  PowerPoint  Game,  or  some  other  negotiated  project.  The  course  meetings 
provide  a forum  for  discussion  of  these  learning  environments  during  the  semester.  Course  readings  that 
introduce  ideas  for  technology  supported  learning  environments  and  experiences  are  also  included. 

Rating:  5 

EDIT  6150  online  has  students  design  three  different  learning  environments.  Those  three  assignments 
are  a cognitive  tool  lesson,  a WebQuest,  and  an  open-ended  project  that  is  similar  to  the  final  project  of  the 
face-to-face  class  above.  Each  student’s  WebQuest  and  open-ended  project  idea  is  discussed  via  the  discussion 
threads  used  as  a part  of  the  course. 

Rating:  5 

Teaching,  learning,  and  the  curriculum 

Teachers  implement  curriculum  plans  that  include  methods  and  strategies  for  applying  technology  to 
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maximize  student  learning. 

EDIT  6150  face-to-face  requires  that  students  develop  their  cognitive  tool  lesson  and  final  project  to  be 
used  with  learners  but  does  not  require  that  they  actually  be  implemented  with  the  target  audience.  Again,  class 
discussion  and  readings  support  the  ideas  needed  to  generate  these  products. 

Rating:  3 

EDIT  6150  online  has  its  participants  implement  the  WebQuest  and  the  open-ended  project  with  at 
least  three  learners  from  the  target  audience.  Evidence  of  implementation  is  shown  by  photo  documentation  of 
the  learners  participating  in  the  lesson.  The  cognitive  tool  lesson  is  planned  so  that  it  could  be  implemented,  but 
evidence  of  implementation  is  not  required. 

Rating:  5 

Assessment  and  evaluation 

Teachers  apply  technology  to  facilitate  a variety  of  effective  assessment  and  evaluation  strategies. 
EDIT  6150  face-to-face  uses  rubrics  for  each  required  assignment.  By  having  students  in  the  class  be  assessed 
by  a rubric,  they  can  see  the  value  of  having  a rubric  for  assessment.  Also,  they  complete  a software  evaluation 
activity  using  a rubric,  which  lurther  illustrates  the  value  of  a rubric.  Additionally,  the  cognitive  tools  lesson 
focuses  on  how  students  can  use  technology  to  assess  and  evaluate  information. 

Rating:  4 

EDIT  6150  online  addresses  this  topic  similarly  to  the  face-to-face  version  of  the  course.  Each 
assignment  has  a rubric  to  which  the  students  in  the  class  are  held  accountable.  Also,  the  cognitive  tools  lesson 
has  the  students  applying  their  knowledge  of  how  different  technologies  can  be  used  for  assessing  and 
evaluating  information. 

Rating:  4 

Productivity  and  professional  practice 

Teachers  use  technology  to  enhance  their  productivity  and  professional  practice.  For  both  of  the 
courses,  the  act  of  enrolling  in  the  course  to  learn  more  about  technology  in  education  demonstrates  their 
willingness  to  enhance  their  professional  practice  through  the  use  of  technology.  However,  each  class  also  adds 
other  components  to  this  topic. 

EDIT  6150  face-to-face  has  a journal  requirement  that  students  must  keep  and  include  in  their  web- 
based  portfolio.  Additionally,  they  include  a description  of  how  each  activity  that  they  complete  as  a student  in 
class  could  be  used  by  their  own  students  in  the  classes  that  they  teach. 

Rating:  5 

EDIT  6150  online  makes  use  of  the  discussion  thread  in  the  online  class  room  in  order  to  facilitate  a 
dialogue  about  how  the  course  topics  relate  to  their  own  practice.  Also,  participants  are  required  to  create  a 
web-based  portfolio  for  the  course  and  in  there  must  have  a reflection  on  each  activity  that  they  have  completed 
as  a result  of  the  course. 

Rating:  5 

Social,  ethical,  legal,  and  human  issues 

Teachers  understand  the  social,  ethical,  legal,  and  human  issues  surrounding  the  use  of  technology  in 
PK-12  schools  and  apply  those  principles  in  practice.  Both  sections  of  EDIT  6150  have  a component  to  the 
cognitive  tools  lesson  and  the  final/open-ended  projects  that  asks  students  to  reflect  on  affordances  that  are 
offered  by  technology  to  the  learning  activity.  Both  classes  also  cover  the  topics  of  social,  ethical,  legal  and 
human  issues  in  assigned  readings  and  discussions. 

Rating:  4 


Time  on  task 

Students  were  asked  to  log  the  amount  of  time  spent  on  a comprehensive  list  of  activities.  Data  entered 
by  the  students  was  saved  into  a database  and  exported  into  a spreadsheet  program  for  analysis  (Table  1).  Each 
log  entry  was  examined  for  consistency.  The  semester  was  composed  of  15  weeks,  so  students  who  entered  less 
than  12  logs  were  not  considered  for  analysis.  Since  each  class  met  once  a week,  it  is  reasonable  to  expect  that 
students  could  miss  a number  of  classes,  or  simply  do  no  measurable  work  for  class  during  a specific  week. 
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Table  1.  Average  times 

on  task  per  student  per 

week  by  category 

Category 

6150  online  (N=l  1) 

6150  f2f  (N=10) 

Average 

Average 

Average 

Average 

Minutes 

Hours 

Minutes 

Hours 

Class  time 

93.73 

1.56 

132.43 

2.21 

Groupwork  online 

3.36 

0.06 

3.23 

0.05 

Groupwork  f2f 

0 

0 

10.97 

0.18 

Individual  work 

125.70 

2.09 

85.27 

1.42 

Message  posting 

28.11 

0.47 

16.15 

0.27 

Technology  problems 

18.38 

0.47 

8.50 

0.14 

Travel 

13.88 

0.31 

42.62 

0.71 

Other 

6.45 

0.11 

3.80 

0.06 

Total 

289.61 

4.83 

302.97 

5.05 

Participants  in  this  study  were  self-selected,  and  as  such,  a generalization  to  the  whole  class  would 
seem  difficult.  Still  the  variance  of  data  is  so  substantial  (online,  N=ll,  M = 4344.09,  S = 2378.85;  f2f,  N=10, 
M = 4544.50,  S = 1724.09)  that  a reasonable  degree  of  confidence  can  be  exercised  when  discussing  the  results 
as  representative  of  the  whole  class.  Were  this  a time  comparison  study,  it  would  be  easy  to  note  that  no 
significance  differences  exist  between  the  courses  in  terms  of  total  time-on-task  per  student  (Table  2). 
Examining  the  specific  categories  provides  a better  context  for  the  examination. 

Table  2.  Total  time  -on-task  per  student 


6150  online  6150  f2f 


Total 

Minutes 

Total 

Hours 

Total 

Minutes 

Total 

Hours 

1787 

29.78 

2110 

35.17 

2445 

40.75 

2542 

42.37 

2458 

40.97 

3313 

55.22 

2480 

41.33 

3665 

61.08 

3440 

57.33 

3980 

66.33 

3745 

62.42 

4755 

79.25 

3980 

66.33 

5465 

91.08 

4660 

77.67 

5970 

99.50 

5585 

93.08 

6150 

102.50 

8290 

138.17 

7495 

124.92 

8915 

148.58 

Data  confirm  some  of  the  traditional  assumptions  regarding  online  courses.  Students  in  the  online 
classroom  traveled  less  than  those  in  the  face-to-face  class.  It  is  not  possible  to  ascertain  whether  travel  time 
was  a defining  factor  in  choosing  the  course  either  of  these  courses.  Still,  since  these  courses  were  offered 
simultaneously,  the  data  suggest  that  travel  time  might  have  been  an  important  factor  in  choosing  the  session  of 
6150  a student  would  take.  It  is  often  assumed  that  more  technology-related  mishaps  will  occur  in  an  online 
classroom,  because  the  computer  and  an  internet  connection  are  needed  for  class  time.  Both  classes  made 
extensive  use  of  computer-related  technologies,  but  there  were  more  technology-related  problems  reported  by 
students  in  the  online  class.  Though  the  time  spent  per  week  it  is  not  in  itself  sizeable,  it  represents  a little  over 
5-percent  of  the  time  students  spent  in  the  online  class.  Comments  indicated  some  frustration  with  the  audio 
connection,  a common  initial  problem  in  the  online  classroom.  Students  are  required  to  download  a java-based 
plug-in  in  order  to  use  the  two-way  audio,  which  often  must  be  tweaked  to  work  behind  a firewall.  As  one 
student  in  the  online  classroom  declared:  “I  find  now  that  "problems  with  technology"  are  just  part  of  the 
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work!” 


Other  observable  differences  in  workload  between  the  students  in  each  course  can  be  attributed  to 
pedagogical  decisions.  Students  reported  spending  less  time  in  class  because  the  online  class  rarely  convened 
for  more  than  two  hours  every  week.  Related  to  this  observation  is  the  finding  that  students  spent  more  time  on 
individual  work  in  the  online  class.  Time  was  incorporated  into  the  classroom  instruction  in  the  face-to-face 
section  of  the  class  for  completion  of  assignments  whereas  students’  only  time  to  complete  individual 
assignments  in  the  online  course  was  on  their  own  time.  Group  work  was  not  written  into  the  curriculum  of  the 
online  class  and  the  time -on-task  data  confirmed  that  students  spent  little  time  cooperating  in  their  projects.  This 
finding  suggests  that  students  will  likely  not  engage  in  group  work  unless  encouraged  by  their  instructor  or 
demanded  by  the  assignment  (Hill,  2002). 


Instructor  reflection 

Examining  the  workload  of  each  student  in  the  classroom  provided  for  an  interesting  observation: 
some  students  dedicated  as  little  as  30  hours  to  the  course,  while  others  set  aside  150  hours,  a five-fold 
difference!  The  low-end  numbers  are  especially  shocking  when  considering  that  class  time  alone  (as  reported  by 
the  students)  would  consume  approximately  1.56  hours  per  week,  for  a total  of  23.4  hours  for  class  time  alone 
during  the  full  15  weeks. 

In  order  to  examine  some  of  the  issues  associated  with  such  disparities,  a follow  up  e-mail  was  sent  to  four 
students:  the  two  who  reported  the  lowest  and  highest  time -on-task  for  each  class. 

Each  student  was  asked  to  confirm  that  they  had  completed  the  logs  correctly,  and  to  provide  an 
explanation  for  the  higher/lower  than  average  workload.  The  two  students  with  the  highest  reported  workload 
(online  = 148.58  hours,  and  f2f  = 124.92  hours)  attributed  the  amount  of  workload  to  their  general  inexperience 
with  computing  and  software  used  in  his  course  (‘‘It  consumed  my  time  because  I am  not  computer  savvy”). 
Both  students  with  the  lowest  workload  also  confirmed  the  estimates.  The  responses  indicate  that  both  students 
were  knowledgeable  about  the  course  content.  One  student  added  that  she/he:  ‘‘tried  to  budget  my  time  very 
wisely”.  The  other  responded  indicated  that  she/he  “basically  knew  everything  covered  in  the  class”. 

Once  the  log  data  was  compiled,  it  was  taken  to  the  course  instructors.  A semi -structured  interview 
was  used  requesting  that  each  instructor  reflect  on  the  time -on-task  data.  The  most  interesting  aspect  of  the 
interview  surrounded  the  large  variance  of  reported  workload.  One  instructor  could  not  believe  that  anyone 
could  have  dedicated  less  than  30  hours  to  his  whole  course,  while  jotting  down  estimates  of  the  number  of 
hours  a student  would  minimally  have  to  dedicate  to  complete  each  project.  The  instructor  estimated  that  a 
student  would  need  at  least  double  the  time  (60  hours)  to  complete  the  assignments  in  this  course.  Even  though 
the  log  confirmed  that  the  student  devoted  only  30  hours  to  the  course,  the  instructor  continued  to  affirm  it  to  be 
impossible.  The  instructor  was  then  told  that  the  student  had  confirmed,  via  email,  that  indeed  the  30  hours  were 
approximately  correct  - moreover,  the  student  had  received  an  “A”  (full  grade”)  in  course.  Even  though 
substantial  evidence  was  presented  to  confirm  the  validity  of  the  data,  the  instructor  did  not  shift  his  opinion 
regarding  the  minimum  workload  required  for  the  course. 

Conclusion 

It  is  clear  from  the  analysis  above  as  would  be  expected  that  both  of  the  courses  address  each  topic 
area  of  the  ISTE-NETS  and  therefore  Georgia’s  PSC  standards.  But  the  differences  lie  in  the  manner  in  which 
each  is  addressed  described  in  the  findings  above.  Additional  distinguishing  features  of  each  individual  course 
are  based  on  the  nature  of  its  delivery  and  the  structure  of  its  curriculum. 

EDIT  6150  face-to-face  has  the  support  of  the  instructors  and  classmates  for  students  as  they  run  into 
technical  difficulties,  which  could  be  helpful  for  those  who  need  personal  attention  when  learning  new  skills 
and  concepts.  Not  all  of  the  skills  that  they  learn  in  this  course  are  directly  tied  to  classroom  application,  and 
can  be  used  more  for  teacher  productivity,  or  even  personal  use.  Since  this  course  is  not  only  offered  to  in- 
service  teachers,  this  course  allows  for  the  opportunity  of  input  from  people  who  are  new  to  the  profession  or  in 
non-teaching  roles. 

While  EDIT  6150  face-to-face  could  meet  the  needs  of  learners  of  all  skill  levels,  the  online  version  of 
EDIT  6150  does  require  some  prerequisite  technical  ability.  At  the  minimum,  a student  in  this  course  should  be 
comfortable  with  basic  web  navigation,  sending  and  receiving  of  emails  with  attachments,  and  a functional 
knowledge  of  Windows  (ability  to  change  screen  settings,  etc.)  Also,  a student  taking  EDIT  6150  should  have 
the  ability  to  work  independently  and  yet  still  contribute  to  the  class  as  a whole. 

In  conclusion,  we  offer  brief  descriptions  of  the  types  of  students  who  might  be  best  served  by  each 
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class  based  on  our  findings.  A student  in  EDIT  6150  face-to-face  may  come  from  various  settings,  but  is  usually 
best  served  by  having  some  connection  to  education  so  that  the  examples  and  suggestions  given  in  class  have 
some  significance.  But  since  none  of  the  lessons  are  required  to  be  implemented  with  students,  it  is  not  essential 
that  students  in  this  course  have  a classroom  connection.  His  or  her  level  of  skill  will  be  irrelevant  since  plenty 
of  in  class  support  is  given  for  those  lacking  skills.  EDIT  6150  online  requires  more  prerequisite  skills  than 
either  of  the  other  courses.  Students  who  prefer  a more  self -directed  environment  would  be  best  served  by  this 
course  and  should  have  an  available  classroom  for  implementing  the  required  assignments. 

The  time -on-task  analysis  provided  for  valuable  confirmation  of  pedagogical  decisions  by  each 
instructor.  It  further  supports  some  of  the  “truisms”  of  distance  education  as  discussed  above  (travel 
preferences,  technology  problems).  The  results  presented  here  further  our  belief  that  time -on-task  data  can  be 
far  more  useful  if  it  is  not  used  for  comparison  purposes.  The  use  of  student  reported  data  can  fruitfully  be  used 
as  a reflection  tool  for  instructors,  in  analyzing  the  actual  student -response  (measure  by  workload)  to 
pedagogical  choices. 
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Abstract 

Student  participation  in  online  learning  activities  is  a growing  priority  for  Australian  government 
schools.  ‘Online  projects’  have  emerged  as  a new  learning  form,  building  on  non  -computer  problem-based 
learning  approaches.  This  paper  reports  on  a study  of  online  learning  projects  implemented  in  classes  from 
year  2 to  year  11.  The  purported  benefits  of  online  learning  were  explored  through  in-depth  case  studies  of  the 
selected  projects.  Results  of  the  study  are  presented,  providing  a window  onto  the  learning  events  as  each 
project  unfolded  and  highlighting  the  achievements  of  students.  The  study’s  findings  have  significant 
implications  for  education  systems  and  teachers,  in  the  design  and  implementation  of  online  projects  as  part  of 
an  effective  learning  provision  for  students.  The  potential  and  limitations  of  an  online  project  approach  are 
placed  in  the  context  of  online  learning  developments  occurring  in  New  South  Wales  and  Australia. 

Introduction 

In  the  New  South  Wales  (NSW)  government  school  system  (Australia’s  largest  schooling  authority),  as 
in  many  other  education  systems,  the  incorporation  of  computer-based  technologies  has  been  a major  priority 
for  the  last  decade,  with  the  familiar  expectation  that  learning  enhancements  would  be  achieved.  Also  familiar 
are  the  disappointing  outcomes  of  computer  use,  in  terms  of  usage  levels  and  evidence  of  effective  learning  or 
in  changes  to  teaching  practice  (Audit  Office,  2000;  Hayes,  Schuck,  Segal,  & Dwyer,  2001;  Roberts  Research 
Group,  2002). 

The  emphasis  on  hardware  provision  and  connection  of  schools  to  the  Internet  needs  to  be  matched 
with  equal  emphasis  on  demonstrating  ways  that  computing  technologies,  particularly  online  technologies,  can 
significantly  add  to  teaching  programs  and  student  achievement,  e-learning  or  online  learning  as  an  alternative 
to  face-to-face  lessons  in  particular  school  circumstances  (small  demand  for  subjects,  teacher  shortage,  distance 
access)  have  become  established  as  niche  services.  Yet  when  faced  with  an  expectation  that  online  learning  will 
be  part  of  all  students’  learning,  we  are  still  asking  the  question  “What  is  online  good  for?”  How  online  learning 
activities  may  provide  new  opportunities,  and  in  what  form,  constitutes  much  of  the  current  debate. 

Online  projects  and  the  Australian  online  landscape 

The  Internet’s  characteristic  feature  is  the  ability  to  be  ‘connected’  - to  information,  people  and 
products.  In  NSW  schools,  currently  most  frequent  use  is  made  of  connection  to  information  (online 
encyclopaedia,  web  searching,  WebQuests)  and  the  associated  learning  activities  (Audit  Office,  2000;  Cooper, 
Jamieson-Proctor,  Crawford,  & Nuyen,  2001;  Wyld  & Eklund,  1997).  There  is  growing  interest  in  educational 
products  that  will  provide  teaching  and  learning  activities  for  direct  use  by  students.  The  largest  proportion  of 
the  investment  by  Australian  education  systems  is  going  into  the  development  of  learning  objects,  which  are 
expected  to  provide  new  opportunities  and  promote  new  ways  of  learning,  as  well  as  filling  current  resource 
gaps.  Success  will  depend  on  the  ability  of  objects  to  model  complex  concepts  and  events  that  are  beyond  the 
scope  of  school  resources,  and  to  provide  self-paced  pathways  through  sequences  of  materials  (Australian 
Education  Systems  Officials  Committee  (AESOC),  2001).  To  date  there  is  little  product  to  show  for  this  and 
even  less  evidence,  worldwide,  for  significant  learning  changes. 

The  ability  to  connect  to  people  provides  a third  area,  with  the  promise  of  adding  new  dimensions  to 
learning  activities,  particularly  in  mainstream,  class-based  environments.  Potential  activities  include  direct  e- 
mail  contact  between  students  and  with  others,  participation  in  online  discussions  or  mailing  lists,  participation 
in  projects  with  students  in  other  places  and  contribution  to  real-world  activities.  Structured  learning  uses  of 
online  communications  often  fall  into  this  latter  group,  presented  in  the  form  of  ‘online  projects’. 

All  Australian  education  systems  contribute  to  the  OzProjects  directory  site  administered  by  EdNA 
Online  (Education  Network  Australia,  2003).  The  site  provides  a central  registry  of  local  and  international 
projects,  including  some  created  by  state  education  systems,  with  links  to  selected  overseas  collections.  It  is 
certainly  not  the  only  source  of  online  projects  being  implemented  in  Australian  schools.  It  does,  however. 
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represent  an  endorsement  of  online  projects  as  e-learning  content  for  school  use,  although  financial  support  of 
the  site  (and  of  online  projects  in  general)  is  minimal,  and  hugely  varied  across  state  education  systems. 

The  study  on  which  this  paper  is  based  explores  how  online  projects,  as  one  form  of  class  activity,  can 
contribute  to  the  learning  outcomes  of  students  and  to  the  effective  use  of  Internet  technologies  in  K-12  school 
settings. 

Activities  describing  themselves  as  online  projects  range  from  simple  information  sharing  or  web 
publishing  activities;  to  those  that  bring  together  problem-based  learning  approaches  and  the  promise  of 
increased  connection  to  and  opportunities  for  collaboration  with  people  and  organisations  beyond  the  school. 
Descriptions  of  project  types  or  categories  (including  simulations)  are  provided  in  several  contexts  (Berenfeld, 
1996;  Ferrari,  Taylor,  & Vanlehn,  1999;  Global  SchoolNet,  2001;  Harris,  2002),  generally  reflecting  differences 
in  the  amount  and  nature  of  the  interaction  between  participants  and  /or  the  complexity  of  the  task.  Different 
learning  opportunities  are  promised  as  students  move  into  a wider  learning  environment,  and  as 
communications  facilities  change  the  way  activities  are  framed  and  undertaken.  Online  projects  are  currently 
implemented  in  pockets  of  schools  and  classes,  both  in  Australia  and  elsewhere,  with  little  consistent  promotion 
or  participation.  The  benefits  of  participation  are  expounded  by  project  providers  (Donlan,  1998  - heavily 
referenced  in  Education  Network  Australia,  2003;  iEARN,  2003)  and  yet  participation  in  projects  remains 
relatively  invisible  in  reports  of  school-based  use  of  ICT  in  Australia  and  in  the  research  literature  around  ICT 
integration. 

Studies  of  online  projects  have  focused  on  an  ‘overview’  approach,  mapping  the  occurrence  and  nature 
of  projects  (Harris,  2002)  or  documenting  individual  cases,  with  an  emphasis  on  implementation  issues  and 
suggestions  for  their  successful  operation  (Carr,  2001;  Wyld  & Eklund,  1997).  While  teachers  recount  their 
positive  experiences  of  participation  in  online  projects  and  the  largely  motivational  benefits  to  students  (for 
example,  (Brunsden,  2000;  Clark,  2002;  Roach,  2004;  Robertson,  2000),  anecdotal  evidence  of  the  value  of 
online  projects  is  not  yet  well  supported  by  research  into  the  experiences  of  learners  or  evidence  of  outcomes 
achieved.  Concerted  research  is  required  that  investigates  and  demonstrates  how  and  what  students  are 
achieving  through  their  participation  in  the  online  activities  being  advocated  (Bennett  & Lockyer,  1999; 
Windschitl,  1998). 


The  Study 

The  study  reported  here  was  conducted  in  four  classes  from  Year  2 to  Year  11,  each  participating  in  a 
project  that  moved  beyond  information  gathering  and  sharing.  The  selected  projects  sought  to  introduce  a 
complex,  purposeful  task  (collaborative  design,  problem  to  solve)  extending  student  activity  beyond  the  school. 
In  each  case  the  project  was  implemented  as  an  integral  part  of  class  activities.  Use  of  online  technologies  was  a 
pivotal  part  of  each  project,  if  not  necessarily  the  aspect  where  most  time  was  spent. 

The  purported  benefits  of  online  learning  were  explored  through  in-depth  case  studies,  allowing  the 
researcher  to  be  part  of  the  class  for  the  duration  of  each  project.  A multi- method  approach  was  taken  to  gain 
insights  into  the  learning  occurring  through  a variety  of  views,  especially  those  of  students.  Data  were  collected 
through  interviews  with  students  and  teachers,  triangulated  with  extensive  observations  of  class  activities  over 
the  duration  of  each  project.1  Analysis  of  student  products  provides  evidence  of  outcomes  achieved,  particularly 
related  to  syllabus  requirements. 

A significant  feature  of  the  study  is  the  value  placed  on  students’  perspectives  of  the  experience  and 
learning  achieved.  Students’  reflections  on  activities  as  they  occurred,  and  in  interviews  at  different  stages  of 
the  project,  are  used  as  a primary  focus  of  analysis;  foregrounding  students’  perceptions  of  the  project 
experience  and  the  (sometimes  less  obvious)  learning  achieved.  Class  teachers  described  how  they  selected  and 
implemented  the  projects.  However,  it  is  their  knowledge  of  students,  revealed  through  interviews  and  informal 
reflections,  that  contributes  most  to  an  understanding  of  the  project  learning  experience:  providing  insights  into 
what  was  occurring  in  the  class  and  what  was  achieved  as  a result. 

Online  projects  in  action 

In  this  paper  I present  two  stories  from  the  intensive  case  studies,  one  located  in  a primary 
(elementary)  setting  and  one  in  a secondary  setting.  A detailed  description  of  each  project  is  provided, 
particularly  highlighting  the  activities  and  responses  of  students,  followed  by  analysis  of  some  significant 
issues.  The  exploration  of  the  particular  contributes  to  a broader  picture  of  online  projects  and  their  place  in 
class-based  learning. 
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Murder  Under  the  Microscope : an  eco-mystery  for  years  5-8 

The  tension  is  palpable.  A final,  collective  decision  needs  to  be  made  about  the  identity  of  the 
ecological  criminal,  drawing  on  the  research,  interpretation  of  clues  and  sometimes  agonisingly  difficult 
reasoning  that  has  taken  place  over  the  last  few  weeks.  A pair  of  students  checks  the  latitude  and  longitude  of 
the  suspected  crime  site.  Another  group  is  arranging  the  issues  summaries  for  the  class’  consideration,  while 
the  bulk  of  the  class  double-checks  the  reasons  why  most  villains  have  been  eliminated  from  suspicion.  They  are 
competing  against  some  2000  other  teams  from  schools  across  Australia  and  beyond,  and  the  deadline  for 
making  an  accusation  is  looming. 

All  the  ingredients  of  a murder  mystery  are  here:  a victim,  dead  in  suspicious  circumstances;  a forensic 
scientist  providing  a complexity  of  test  results;  an  array  of  suspects  each  with  a possible  "modus  operandi’; 
witnesses  and  bystanders  ready  to  give  contradictory  accounts  of  events;  all  presented  by  a world-weary 
investigator  in  the  field.  The  detective  work  was  done  by  students  in  Class  4/5/6  (team  name  Ecostars ),  in  their 
role  as  ‘eco-detectives’,  struggling  to  piece  together  the  information  and  clues  to  lead  them  to  the  solution: 
identifying  the  victim,  the  crime  site,  the  villain  and  probable  cause  of  the  fatality.  Information  was  revealed 
through  weekly  television  broadcasts  (also  available  online)  and  regular  updates  provided  on  the  project  web 
site.  The  second  phase  of  the  project  involved  students  in  the  development  of  a rectification  plan  for  the  affected 
site,  with  the  intention  of  preventing  similar  ‘crimes’  from  occurring. 

The  project  was  implemented  over  a nine  week  period  with  three  to  five  hours  of  class  time  spent  each 
week.  Outcomes  in  Science  and  Technology,  English  and  Environmental  Education  were  anticipated. 

Capitalising  on  early  enthusiasm  for  the  project,  the  class  spent  the  first  weeks  working  in  groups  to 
explore  the  ten  possible  crime  sites,  two  per  group.  They  identified  the  information  needed  for  each  site, 
negotiating  the  fields  of  the  database  that  became  the  enduring  shared  resource  and  reference  point  throughout 
the  investigation.  A similar  process  of  distributed  investigation  was  used  a few  weeks  later,  to  develop 
understandings  about  the  catchment  issues  that  might  be  related  to  the  crime.  Each  group  selected  and 
completed  structured  activities  provided  by  the  project. 

The  whole  class  eagerly  awaited  the  first  broadcast  from  Catchment  HQ  that  revealed  the  crime 
scenario  and  the  20  potential  victims  and  villains,  and  began  the  stream  of  information  and  clues  to  be  sifted, 
interpreted  and  related  to  the  background  information  they  had  compiled.  Immediately,  students  were  able  to 
eliminate  a number  of  victims,  villains  and  sites.  They  debated  the  impact  of  the  new  information,  providing 
evidence  from  their  group  investigations,  with  minimal  intervention  from  the  teacher.  Subsequent  broadcasts 
were  characterised  by  absolute  quiet  in  anticipation,  followed  by  extraordinary  levels  of  attention,  even  when 
frustration  was  expressed  about  other  aspects  of  the  investigation. 

Interaction  with  the  project  web  site  was  a defining  feature  of  this  project.  Accumulated  information 
was  stored  in  various  areas,  games  and  activities  were  provided  for  enjoyment  as  well  as  providing  additional 
clues,  and  the  site  provided  direct  access  to  reference  materials,  one-to-one  communication  with  the  scientist 
experts,  and  links  to  outside  information  sources.  Information  is  deliberately  delivered  progressively 
throughout  the  investigation,  allowing  students  time  to  gather  ideas  and  build  understanding,  before  each  new 
set  of  data  and  clues  arrived.  New  data  acted  as  a learning  reward,  sustaining  interest  as  predictions  and 
decisions  made  by  students  were  confirmed  or  refuted.  The  project  itself  guided  the  investigation,  steering 
students’  deductive  efforts.  Additional  input  was  delivered  via  the  Crime  Scene  Reports  and  daily  Newsflashes, 
which  were  impatiently  downloaded  each  morning.  Disappointment  was  strong  when  they  did  not  reveal 
something  obviously  useful. 

Whole  class  discussion  was  used  to  share  ideas  across  the  groups.  On  some  days  this  worked  better 
than  others,  as  children  found  it  hard  to  make  the  connections  between  disparate  sources  and  bites  of 
information  - a major  difficulty  identified  by  the  teacher.  By  the  middle  of  the  investigative  phase  the  class  hit 
a trough  in  enthusiasm.  A growing  concern  for  the  class  teacher  was  the  feeling  that  many  students  were  not 
productively  involved,  in  the  group  activities  or  in  open  discussions.  Ongoing  technical  difficulties  with  the 
class  computers,  though  relatively  minor,  caused  added  tensions  as  groups  were  delayed  in  pursuing  their 
research  or  had  to  use  computers  located  in  other  areas  of  the  school. 

The  momentum  picked  up  again  with  each  broadcast,  particularly  as  the  deadline  for  accusations  drew 
nearer.  Gaps  in  understanding  were  being  filled  and  more  was  revealed  through  the  drama,  enabling  the  class  to 
make  more  confident  assertions  and  narrow  the  scope  of  the  follow  up  questions  to  be  investigated.  By  this 
stage  the  difficulties  of  group  operation  had  largely  been  resolved  and  the  time  spent  on  scaffolding  group 
organisation  was  paying  off.  The  class  teacher,  Hannah  felt  happy  to  leave  the  investigative  work  to  the 
students,  spending  more  of  her  time  assisting  the  groups  to  keep  functioning  and  structuring  activities  to 
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maintain  involvement  of  individual  children. 

By  week  5 the  enthusiasm  was  tangible.  The  crimes  site  and  victim  lists  were  each  narrowed  down  to 
two  possibilities,  with  specific  research  questions  being  allocated  to  pairs  of  children  to  follow  up.  Questions 
were  posted  to  the  project  scientists  and  while  the  answers  were  of  limited  value  in  resolving  the  issue,  some  of 
the  questions  posted  by  other  teams  provided  useful  additional  ideas.  Sifting  through  the  hundreds  of  questions, 
however  reduced  this  to  a matter  of  luck  ratherthan  systematic  analysis.  Several  students  took  the  unexpected 
step  of  contacting  the  Department  of  Water  Conservation  by  phone,  and  succeeded  in  getting  an  answer  to  their 
question. 

Deadlines  for  individual  tasks  were  used  to  keep  groups  on  track  and  maintain  the  urgency.  In  a frenzy 
of  activity  on  the  day  of  the  accusation,  the  class  arrived  at  a tentative  solution  to  the  mystery.  There  was  still 
uncertainty  about  the  cause  or  ‘issue’  involved.  The  competitive  aspect  kicked  in  at  this  point  and  the  class  was 
divided  about  whether  to  submit  their  accusation  immediately  (only  one  accusation  was  allowed  per  team,  and 
the  first  correct  solution  would  ‘win’)  or  wait  for  the  final  broadcast  to  provide  any  final  clues  and 
confirmations. 

The  solution,  and  ‘correct  answer’,  was  posted  on  the  web  site  the  next  day.  The  Ecostars  did  not  get 
all  four  elements  correct,  having  made  an  error  in  locating  the  crime  site  latitude  and  longitude  and  in 
incorrectly  identifying  the  environmental  issue.  Given  the  level  of  involvement  throughout  the  project,  it  was 
surprising  how  rapidly  the  immediate  let  down  of  not  getting  the  answer  right,  gave  way  to  further  thinking 
about  the  solution.  There  could  have  been  recriminations  over  the  error  made  by  the  pair  who  checked  the  grid 
references.  While  the  teachers’  distress  was  obvious  as  she  felt  responsible  for  not  having  made  a final  check, 
the  children  were  surprisingly  understanding.  Attention  quickly  moved  to  the  other  error  in  their  solution:  the 
wrong  issue.  Rather  than  accept  they’d  ‘just  got  it  wrong’,  a lively  debate  ensued  about  the  relative  merits  of  the 
‘correct  answer’  in  comparison  to  the  class’  decision.  While  this  may  be  interpreted  as  justification  (or  just 
plain  sour  grapes)  the  insights  and  arguments  put  forward  demonstrated  a significant  level  of  understanding  of 
the  both  the  issue  that  was  suggested  and  the  others  that  had  been  considered  along  the  way.  Several  groups  of 
children  were  able  to  outline  and  substantiate  how  their  preferred  solution  reflected  the  clues  they  had  been 
provided.  A similar  debate  was  played  out  on  the  teachers’  forum  of  the  project,  with  a number  of  dissenting 
arguments  put  forward.  The  outcome  for  both  teachers  and  students  was  an  acknowledgement  of  the  interplay 
of  influences  and  the  cumulative  effect  of  multiple  environmental  pressures  - a sound  learning  outcome  in 
itself. 

Bring  Modern  History  to  life:  the  Middle  East  simulation 

The  room  is  full  of  variously  dressed  ‘conference  delegates’ ; passionately  arguing  their  points,  leaping 
out  of  seats;  or  slumped,  resignedly  feeling  the  frustration  of  impasse.  A quickly  scribbled  note  passes  between 
delegates,  framing  up  a response  to  an  accusation  or  proposal.  Microphone  in  hand,  the  press  asks  questions 
that  make  Arafat  squirm,  al-Assad  leap  to  her(his)  feet  and  George  Bush  defer  to  his  co-delegates  for  support. 

The  Middle  East  simulation  immerses  Year  11  Modern  History  students  in  the  issues  and  personalities 
of  Arab-Israeli  politics.  Originally  developed  by  the  Macquarie  University  Centre  for  Middle  East  and  North 
African  Studies,  for  use  by  tertiary  students  studying  the  politics  of  the  wider  Middle  East  region  (Macquarie 
University  Centre  for  Middle  East  and  North  African  Studies,  2003),  the  simulation  has  been  adapted  to  meet 
the  needs  of  high  school  participants  focusing  specifically  on  the  Arab-Israeli  conflict. 

In  groups  of  three,  students  take  on  roles  of  significant  characters  in  contemporary  Middle  East  affairs, 
using  Internet  technologies  to  interact  and  play  out  the  action  in  a likely,  if  not  real,  political  scenario.  The 
participants  were  60  students  in  three  classes,  across  two  Sydney  high  schools.  Several  ‘control’  roles  were 
created  to  allow  monitoring  and  assessment  of  activities  by  class  teachers  and  ‘controllers’  from  the  Centre  for 
Middle  East  and  North  African  Studies.  The  controllers  provided  support  and  guidance  to  students,  as  required, 
allowing  them  to  draw  on  the  expertise  of  the  participating  university  faculty  members. 

Following  the  release  of  a scenario  wh  ich  sets  the  scene  for  the  simulation  to  follow,  the  action  and 
reaction  was  driven  by  the  ideas  and  decisions  of  students,  unfolding  over  a three  week  period.  The  culmination 
of  the  project  was  a Conference  Day  where  players  came  together,  face-to-face,  to  negotiate  around  key  issues 
in  the  conflict.  All  the  lead  up  action  took  place  online:  messages  were  sent  by  e-mail,  players  used  chat 
sessions  to  negotiate  in  real  time  and  the  simulation  web  site  provided  information  and  facilities  to  help  students 
explore  issues  and  plan  their  participation. 

The  overall  response  of  students  to  this  very  different  learning  environment  was  unexpectedly  varied. 
Problems  with  groups  and  difficulty  feeling  ‘in  role’  made  it  less  productive  for  some  students,  and  the  timing 
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of  the  project  was  the  subject  of  frequent  complaints.  Yet  in  only  a very  few  cases  were  these  barriers  sufficient 
to  dampen  enthusiasm  or  outweigh  the  benefits  described  by  students. 

Levels  of  involvement  were  extremely  high,  evidenced  in  all  sorts  of  ways  - from  heated  arguments  on 
the  school  bus,  or  reassurances  between  friends  after  impassioned  conference  debates,  to  numbers  of  players 
who  continued  to  log  on  to  the  site  well  after  the  simulation  was  finished.  This  is  not  to  suggest  that 
participation  was  easy.  At  times  the  challenge  to  existing  ideas  was  difficult  to  deal  with,  particularly  for  those 
students  who  held  strong  views  prior  to  the  simulation. 

Because  I 'm  Jewish  and  I was  playing  Syria. . . when  you  went  inside  their  views  it  gives  a different 
perspective  on  the  whole  situation...  I had  never  seen  or  even  thought  about  other  countries  and  what  they 
are  thinking...,  another  perspective,  never  thought  about  it  before... 

Overwhelmingly,  interviewed  students  appreciated  being  able  to  direct  the  action  themselves;  having 
to  weigh  up  ideas,  think  strategically  and  develop  careful  negotiating  skills. 

...it  was  great  - not  being  checked  all  the  time  - do  what  you  want  in  your  character  - you  ’ re  in  control  of 
the  character. 

The  project  was  time  consuming,  and  the  ‘every  night’  commitment  was  problematic  when  other 
priorities  were  neglected.  Several  students  described  it  as  engrossing,  addictive  and  themselves  as  ‘becoming 
obsessed’.  Strategies  had  to  be  developed  by  students  to  manage  participation.  Working  to  a team  schedule, 
setting  time  limits  and  flexibly  sharing  the  workload  to  accommodate  other  demands  on  team- members’  time, 
were  all  used  to  deal  with  time  pressures. 

'Being  the  character’  exerted  a pressure  to  do  the  research  and  develop  a deep  understanding  of  the 
role  being  played.  In  order  to  take  strategically  consistent  actions,  students  developed  understandings  of  the 
range  of  characters,  not  just  their  own.  The  project  shifted  the  emphasis  from  learning  about  events  and 
consequences,  to  experiencing  the  processes  of  policy  making,  tactics  and  making  difficult  decisions  at  different 
levels  of  politics. 

...there  is  like  internal  and  external  results  of  things  - all  the  people  who  die,  poor  conditions  and  then  the 
political  side...  thinking  as  the  nation....  It’s  really  hard  to  know  which  to  do,  because  if  you  just  go  from 
the  side  of  the  civilians  it’s  like  giving  up  your  nation ’s  rights  - like  your  beliefs  ... 

The  conference  day  was  characterised  by  strongly  expressed  positions,  impassioned  responses  and 
heart -felt  attempts  to  find  solutions.  The  depth  of  knowledge  and  empathy  with  their  character’s  position  and 
outlook,  developed  through  the  online  activities,  allowed  students  to  confidently  argue  their  points  and  respond 
‘in  character’.  Students  developed  a strong  sense  of  the  complexity  of  the  situation  and  reasons  behind  it,  all  the 
while  maintaining  a remarkable  optimism. 

Discussion  of  findings  from  the  two  cases 


Variations  on  a theme 

I have  chosen  to  present  accounts  of  these  two  online  projects  because  they  are  so  very  different:  in 
learning  area,  in  nature  of  project  approach,  in  technologies  used  and  in  age  of  participants.  This  selection  of 
cases  is  not  proposed  as  representative  in  any  way  nor  do  I want  to  suggest  that  they  represent  polar  opposites  or 
even  points  on  a continuum.  They  do  demonstrate  the  variation  that  exists  in  the  range  of  projects  that  can 
conceivably  be  developed.  There  is  value  in  exploring  different  settings,  ages  and  learning  area  contexts  as  well 
as  the  different  types  of  project  on  offer.  Multiple  cases  assist  in  building  a more  extensive  view  of  the 
attributes,  value  and  difficulties  that  online  projects  offer. 

What  unites  them  (and  the  others  in  the  study)  is  their  existence  as  constructed  learning  environments. 
They  are  ‘packaged’  as  complete  units  of  work,  offering  complex,  problem-solving  challenges.  They  each  do 
more  than  simply  ‘connecting’  students  in  order  to  communicate  per  se.  They  reflect  commonly  described 
characteristics  of  project-based  learning  approaches  (Katz,  2000;  Moursund,  2002;  Stepien  & Gallagher,  1993; 
Thomas,  2000):  being  implemented  as  a central  part  of  the  class  curriculum,  promoting  increased  student 
autonomy,  engaging  students  in  constructive  investigation  around  concepts  of  significance  through  realistic,  non 
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school-like  topics,  tasks  or  challenges. 

The  experience  of  being  part  of  these  vibrant  learning  spaces  is  highly  seductive  - even  second  hand  as 
I observed  online  the  daily  performances  in  the  Middle  East  simulation  and  recorded  the  student’s  accounts  of 
their  experience.  It  would  be  easy  to  take  away  a glorified  view  of  the  project  experience.  The  purpose  of  this 
paper  is  to  make  a more  critical  examination  of  the  nature  of  the  learning  achieved  and  the  role  played  by  online 
presentation  of  the  projects. 

Reflecting  on  the  learning  experience 

At  the  heart  of  each  project  is  the  expectation  that  students  will  develop  disciplinary  knowledge.  In 
both  cases  the  teachers’  decisions  to  take  part  was  dependent  on  the  project  providing  a learning  challenge  that 
fitted  their  existing  intentions,  directly  related  to  the  required  learning  outcomes.  Analysis  of  activities  and  work 
products  provided  substantial  evidence  for  attainment  of  subject-based  knowledge  that  clearly  met  the  required 
course  or  syllabus  outcomes.  Teachers  in  both  projects  highlighted  that  the  students  did  much  more  than  simply 
reach  these  outcomes. 

The  history  students  themselves  were  keen  to  talk  about  their  wider  learning,  highlighting  an 
appreciation  of  both  sides  of  the  conflict  as  perhaps  the  most  valuable  outcome  of  their  involvement.  New 
depths  of  understanding  of  the  complexity  of  issues  were  described,  with  an  awareness  of  motivations  and  the 
different  points  of  view  that  are  inevitably  present  in  any  conflict.  Building  empathy  with  the  ordinary,  as  well 
as  not-so-ordinary  people  on  both  sides  of  the  conflict  was  a surprising  result  for  some.  One  girl  described  her 
realisation  that  “this  isn’t  about  countries  - they  were  people’’  in  terms  of  it  being  a revelation. 

So  it’s  like  learning  in  3D  - because  it’s  not  just  like  this  is  this  and  this  is  that,  it’s  like  ‘maybe’  - and 
there  is  also  this  side  - different  ways  of  looking  at  it!  (Student  interview,  Middle  East  simulation) 

The  most  significant  feature  of  both  projects  was  the  change  in  the  way  the  learning  was  achieved.  The 
projects  set  up  learning  spaces,  physical  and  conceptual,  that  differed  markedly  from  those  usually  encountered 
by  the  classes.  Activity  shifted  from  "finding  out’  about  events  and  consequences  in  a more  traditional  content 
driven  approach,  to  one  that  required  students  to  participate  in  the  processes  of  investigating,  making  decisions 
and  developing  solutions.  The  range  of  outcomes  was  extended,  providing  a greater  emphasis  than  usual  on  the 
learning  processes  of  the  respective  disciplines. 

. . . obviously  we  learnt  about  the  conflict  - but  more.  You  know  it 's  so  easy  to  stand  back  and  criticise  the 
way  that  politics  work.  Everyone  is  so  stubborn...  it’s  so  much  harder  to  be  so  neat  about  it  now  - to 
criticise  when  you’ve  been  there...  so  I think  it  made  us  realise  it’s  not  that  easy  - these  feelings  have  been 
held  for  years.  You  can’t  just  change  it 

It  was  an  experience,  not  just  a lesson,  or  an  essay...  ( Student  interviews,  Middle  East  simulation ) 

A major  aim  of  the  Middle  East  simulation  (subsequently  referred  to  as  ME  simulation)  was  to  connect 
students’  experiences  to  real  events  in  the  outside  world;  engaging  them  in  the  processes  of  international  politics 
as  well  as  historical  inquiry.  The  simulation  necessarily  required  students  to  ‘find  out’  but  with  an  increased 
depth  of  inquiry  because  they  need  to  take  action  - they  need  to  do  something  meaningful  with  the  information. 

In  solving  the  Murder  mystery,  students  engaged  in  an  investigative  process  that  required  them  to 
gather  and  share  information,  think  carefully  about  relationships  between  pieces  of  data,  make  links  and  see 
casual  relationships,  and  substantiate  ideas  through  reference  to  data  provided  or  information  gathered.  This 
represented  a significant  challenge  to  the  students’  usual  ways  of  demonstrating  learning  and  produced  some  of 
the  greatest  tensions  in  the  project.  Outside  the  project  context,  the  students  were  (and  are)  most  commonly 
required  to  locate  and  select  information  relevant  to  a question  or  topic,  with  repackaging  of  the  information 
sufficing  as  a demonstration  of  learning.  The  messy  process  of  looking  for  evidence  and  then  testing  it  against 
established  understandings  and  other  information  sources,  was  new  and  difficult  for  many  students.  The 
contribution,  however,  to  achievement  of  learning  processes  outcomes  of  Science  and  Technology2'  particularly 
investigation,  was  a major  benefit  of  the  project. 

Really  useful  learning  problems  are  not  easy  to  solve.  Significance  relies  on  the  problem  reflecting  real 
world  conditions:  in  these  cases,  being  contentious,  complicated  by  multiple  viewpoints  and  vested  interests. 
Both  projects  created  learning  environments,  one  online  the  other  in  the  classroom,  conducive  to  knowledge 
building:  problematising  the  topics,  relying  on  students  to  do  the  intellectual  work,  while  supporting  them  in 
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learning  how  to  do  this  (Engle  & Conant,  2002). 

Much  more  was  demanded  of  students;  they  could  not  rely  on  information  retrieval  or  simple,  literal 
readings  of  reference  materials.  They  were  compelled  to  identify  what  they  needed  to  know  and  the  questions 
they  needed  to  ask  in  order  to  be  able  make  the  next  move  toward  the  solution.  The  project  structures  supported 
the  teachers  in  shifting  cognitive  responsibility  to  students  (Scardamalia,  2002),  changing  their  own  roles  and 
those  of  students.  The  ME  simulation  places  all  responsibility  of  the  collective  learning  on  the  students. 

Teachers  provided  support  only  when  specifically  sought  out  by  students,  or  in  extreme,  and  rare  instances 
where  they  need  to  intervene  (although  this  was  not  required  during  the  2003  implementation).  Such  enhanced 
agency  is  particularly  rare  in  senior  secondary  classes,  where  examination  pressure  often  causes  teachers  to 
revert  to  highly  transmissive  pedagogies  in  order  to  ‘cover  the  content’,  but  where  it  might  be  most  urgently 
required  (Heath,  2003). 

Not  knowing  ‘the  answer’  was  also  particularly  significant.  It  was  critical  to  present  an  open-ended 
scenario,  allowing  students  to  work  through  the  processes  of  decision  making,  negotiation  and  compromise 
without  a predetermined  solution  available.  To  truly  engage  in  the  processes  of  diplomacy,  the  characters  had  to 
have  options,  make  choices  and  take  risks  - and  deal  with  the  uncertainty  of  how  others  might  respond. 

In  Murder  the  teacher  remained  a necessary  part  of  the  learning  collective,  taking  a shared  role, 
assisting  regularly  as  needed.  The  difference  in  teacher  role  was  again  assisted  by  not  knowing  the  answer.  She 
was  unable  to  shape  the  direction  of  the  investigation,  even  inadvertently.  Groups  were  held  accountable  for 
contributing  to  the  success  of  the  investigation  and  were  required  to  make  the  knowledge  generated  available 
and  accessible  to  the  rest  of  the  class.  Hannah  was  more  than  a just  a co-learner,  being  responsive  to  students 
cognitive  and  social  learning  needs.  The  more  challenging  nature  of  the  task  revealed  skill  gaps  that  had 
previously  been  hidden,  exposing  assumptions  the  teacher  had  made  about  individual  and  group  competence. 

...[needed  to]  plan  lessons  for  the  kids,  for  those  that  need  them.  The  others  can  sort  through  but  some 
need  more  directed  activities  to  get  them  to  be  able  to  deal  with  it  - some  are  suffering  from  info  overload 
and  are  opting  out.  (Teacher  comment  during  lesson  observation) 

In  addition  to  scaffolding  group  operation  Hannah  realised  she  needed  to  provide  targeted  lessons, 
such  as  guided  deconstructions  of  texts,  modelling  of  question  generation  and  even  basic  information  skills. 

The  importance  of  group  activity  and  the  difficulties  it  presented  strongly  influenced  the  experiences  of 
individuals  in  both  projects.  Worthwhile  skills  and  strategies  for  working  in  groups  were  developed  by  many  in 
the  ME  simulation,  to  manage  workload  and  organise  collective  contributions.  Where  the  groups  worked  well, 
they  added  to  the  building  of  understanding  and  confidence  in  the  subject  matter.  The  group  helped  individuals 
to  work  through  challenging  new  ideas,  to  utilise  or  develop  different  strengths  and  specialised  knowledge 
areas,  and  to  collaborate  in  the  construction  of  responses.  These  effects  were  reliant  on  the  effective  functioning 
of  the  group.  Students  were  quick  to  point  out  when  others  ‘could  hide  behind  the  group’.  While  the  variation  in 
contribution  to  the  shared  task  was  noted  and  reflected  in  the  final  assessments,  there  was  little  way  of 
alleviating  the  added  burden  felt  by  those  whose  groups  did  not  function  well. 

The  amount  of  research  required  and  the  complexity  of  the  information  to  be  digested  throughout  the 
Murder  investigation  necessitated  a division  of  labour  and  the  pooling  of  ideas  and  knowledge  acquired  or 
created.  It  was  clear  that  the  class  was  not  used  to  working  in  this  way.  Early  on,  some  students  openly 
discussed  how  hard  it  was,  but  equally  how  enjoyable  they  found  it.  Others  were  less  enthusiastic,  and  because 
of  the  group  work  structure,  they  found  it  relatively  easy  to  hang  back,  providing  minimal  input  to  group  tasks 
and  avoiding  contribution  to  broader  discussions.  Over  time  a balancing  effect  was  noticed.  Students  had  to 
trust  that  information  being  provided  by  other  groups  was  accurate,  with  the  reciprocal  effect  of  creating  an 
imperative  for  groups  to  produce  worthwhile  contributions  to  the  shared  information  pool.  As  the  project 
progressed  this  process  was  taken  more  seriously.  The  pressure  to  make  a contribution,  in  the  knowledge  that  it 
might  be  the  pivotal  piece  of  information,  resulted  in  a greater  willingness  to  complete  activities  and  share 
findings.  Improved  learning  relationships  involving  trust  in,  and  respect  for  other  class  members,  developed 
during  the  project.  For  Hannah,  the  nagging  doubt  persisted,  however,  about  how  much  had  been  achieved  by  a 
(small)  number  of  class  members. 

Scardamalia  and  Bereiter  (1991)  remind  us  not  to  romanticise  the  idea  of  students  as  independent  learners, 
acknowledging  the  role  of  authoritative  sources  of  various  types.  The  role  that  the  projects  played  in  supporting 
students  in  this  way  is  also  linked  to  ideas  about  authenticity. 

Enhancing  authenticity  is  a claim  often  made  by  advocates  of  ICT  in  learning,  particularly  online 
projects  (Bransford,  Brown,  & Cocking,  2000;  Donlan,  1998;  GLOBE,  2004;  Stepien  & Gallagher,  1993). 
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While  there  is  considerable  diversity  in  the  ways  that  authenticity  is  conceptualised  in  educational  research, 
there  was  an  underlying  assumption  on  the  part  of  teachers  in  the  study  that  the  projects  themselves  were 
‘authentic’  or  true  to  the  situation  they  purport  to  represent,  providing  students  with  valid  insights  and 
understanding  of  the  wider  world. 

As  occurs  in  many  online  projects  (Childnet  International,  2002;  CIESE,  1998-2001;  Global  SchoolNet,  2001) 
the  ME  simulation  and  Murder  both  offered  students  access  to  experts.  Expert  involvement,  however,  was  not 
limited  to  answering  individual  questions.  The  projects  gained  authority  by  their  very  design,  being  developed 
by  specialists  in  the  field  who  ensured  that  ‘the  science  was  right’  or  that  the  ‘likely  if  not  real’  scenario  of  the 
ME  simulation  was  authentic  to  contemporary  events.  As  one  designer  involved  in  Murder  commented,  “it  may 
be  fiction  but  it  can  never  be  fictional”. 

Students’  construction  of  knowledge  was  guided  throughout  the  Murder  investigation  via  the 
progressive  revelation  of  information,  all  of  which  was  subsequently  available  on-demand  on  the  web  site.  The 
questions  posed  by  all  participating  teams  were  available  to  all  as  a further  resource. 

The  university-based  controllers  in  the  ME  simulation  performed  a dual  function:  providing  direct 
responses  to  student -initiated  questions,  and  endorsing  proposed  ‘major’  actions  before  they  happened 
(offensive  strikes,  dismantling  of  a refugee  camp).  Their  advice  assisted  students  in  understanding  and 
interpreting  their  character’s  actions  and  reactions  and  helped  them  to  think  through  alternative  types  of  action 
they  may  take,  without  diminishing  students’  decision  making  ability. 

Control's  reply  to  a request  from  the  CIA  to  leak  a false  report  regarding  the  death  of  Sheikh  Nasrallah,  in 
the  hope  of  driving  people  to  the  negotiating  table: 

George, 

Put  down  the  matches  and  the  petrol...  Attacks  tend  to  move  parties  away  from  the  negotiating  table  rather 
than  towards  it  (thus  the  I-P  peace  process  is  constantly  derailed  by  attacks)  - you'll  find  that  talks  are 
most  likely  when  both  sides  are  exhausted  by  violence. 

If  you  like,  you  can  still  mail  Fox  News  and  make  up  a false  report,  but  it  may  be  counter-productive . . . As 
Director  of  the  CIA,  you  have  vast  resources  and  experience  in  force  management  so  get  out  there  and 
start  managing  these  parties  by  improving  security  on  all  sides. 

Teachers  were  enormously  appreciative  of  the  addition  to  their  resource  repertoire  and  valued  highly 
the  opportunity  for  their  students  to  learn  from  external  specialists.  Students  not  only  benefited  from  the  direct 
input  provided,  but  saw  the  involvement  of  real  scientists  and  real  academics  as  validating  the  work  they  were 
doing.  They  knew  their  learning  mattered. 

In  both  cases  the  projects  were  developed  for  school  use  through  partnerships  between  the  disciplinary 
experts  (government  departments  and  university  academics)  and  educationalists.  This  also  works  as  quality 
assurance  for  teachers  considering  embarking  on  a project-based  activity.  The  ongoing  partnerships  have 
worked  to  make  the  projects  more  than  one-off  events,  being  open  to  an  ever-expanding  number  of  schools  and 
classes  and  elevating  them  to  a level  beyond  many  of  the  information-sharing  projects  that  dominate  the  online 
project  landscape. 

The  two  projects  discussed  here  had  particular  strengths  in  adding  authenticity  in  terms  of  processes 
and  content.  In  the  other  cases  in  my  study,  those  not  elaborated  in  this  paper,  the  tasks  were  not  as  reflective  of 
real-world  activities.  Ho  wever  they  provided  a greater  level  of  authenticity  and  value  to  students  because  of  the 
audience  for  whom  they  were  completed. 

The  audience  for  activity  in  both  Murder  and  ME  simulation  was  also  extended  beyond  the  class 
teacher.  Participation  in  ME  was  assessed  by  the  project  controllers,  as  well  as  the  class  teachers,  based  on  how 
‘true  to  character’  students  were  in  their  interpretation  of  the  scenario  and  the  actions  proposed.  Working  with 
another  school  added  to  the  effectiveness  of  the  simulation.  Submitting  the  solution  to  the  Murder  mystery  to 
‘Catchment  Headquarters’  provided  an  acute  motivating  effect  and  significant  value  to  the  learning.  In  both 
projects,  students  talked  of  connections  made  to  their  own  interaction  with  and  enhanced  understanding  of 
current  events. 

However,  in  neither  of  these  situations  did  the  activities  and  products  of  the  projects  have  a real  impact 
on  events  or  people  outside  the  class.  This  is  the  obvious  limitation  of  simulations,  which  by  definition  are 
imitations  of  real  events.  But  it  may  help  to  explain  the  disappointing  participation  on  the  second  phase  of 
Murder.  For  the  majority  of  teams,  including  my  case  study  class,  the  solution  to  the  crime  is  the  culminating 
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event  of  the  project.  The  second  phase  involves  the  development  of  the  catchment  plan,  that  brings  together  the 
understanding  students  have  developed  through  the  investigation  and  asks  that  they  put  their  knowledge  into 
practice.  Consistently,  only  around  10-15%  of  teams  continue  to  phase  two  (Interview  with  project  designer) 
despite  the  promotion  of  this  as  the  most  important  aspect,  deserving  of  the  “highest  honour”  (Teachers’ 
Handbook,  p.4). 

How  much  of  these  effects  were  the  result  of  the  projects  being  presented  online? 

Examination  of  the  impact  of  online  presentation  of  the  projects  yielded  some  expected  benefits, 
consistent  with  the  experience  in  other  web-based  activities.  Working  in  the  online  space  created  in  the  ME 
simulation  increased  participation  of  all  students  (Sherry  & Bilig,  2002),  access  to  expertise  beyond  the  school 
and  extended  audience  for  student  activity. 

The  motivating  effects,  so  frequently  attributed  to  online  activities  was  of  less  intrinsic  value  than  the 
other  aspects  of  the  project  environments  already  discussed.  For  most  students  in  the  study  computer  use  is  not 
a novelty  to  be  valued  for  its  own  sake.  On  the  contrary,  Hannah  encountered  continuing  resistance  by  a few 
class  members  to  the  computer-based  activities,  that  was  barely  altered  by  participation  in  the  project.  Once 
again  the  group  work  focus  enabled  students  to  opt  out  of  this  aspect  of  the  tasks. 

Even  in  the  ME  simulation,  where  most  of  the  action  took  place  online,  the  learning  was  not  primarily 
about  using  the  computers.  For  some  it  was  the  first  time  they’d  really  used  e-mail,  but  this  was  rare.  The  online 
environment  of  the  simulation  was  significant  for  several  students  who  for  the  first  time  saw  some  purpose  in 
computer  use  for  school.  Others  were  critical  of  the  interface  because  of  their  extensive  personal  experiences. 

While  the  projects  certainly  put  the  technologies  to  use  in  meaningful  ways,  they  were  not  identified  as 
significantly  improving  students’  computing  skills,  except  in  a few  isolated  cases.  Rather,  they  shifted  the  role 
of  the  technology,  making  it  secondary  or  ancillary  to  the  purpose  and  intention  of  the  learning.  Presenting  the 
projects  online  not  only  adds  to  the  realism  of  the  experience  but  extends  students’  technology  activities  to 
higher  order  uses. 

A strong  message  came  from  the  teachers.  Several  were  tentative  computers  users  prior  to 
implementing  the  project.  The  experience  has  demonstrated  a more  meaningful  use  of  ICT  in  their  classes  and 
provided  suggestions  for  new  ways  of  working.  While  this  doesn’t  mean  they  are  instant  converts,  they  are  now 
looking  for  other  times,  places  and  ways  of  creating  similar  learning  spaces. 

...  it  gave  me  another  view  on  how  to  do  it  and  how  to  use  the  Internet...  I wouldn  't  have  done  it  that  way, 
it  wouldn  ’1  have  occurred  to  me.  ( Teacher  interview  - Murder  Under  the  Microscope) 

Unexpected  effects  also  emerged,  related  to  the  unfolding  of  events,  the  learning  supports  provided 
through  the  project  infrastructure,  and  the  positioning  of  students  that  enhanced  the  authenticity  of  the  task. 

The  murder- mystery  metaphor  of  Murder  Under  the  Microscope  creates  the  drama  and  excitement  of 
the  project.  The  importance  of  the  unknown  result  has  already  been  discussed.  The  progressive  unfolding  of 
events  was  only  possible  through  the  delivery  of  the  materials  online  and  through  the  broadcasts.  Daily  and 
weekly  inputs  not  only  maintained  the  momentum  of  the  investigation  over  several  weeks,  but  also  helped 
students  cope  with  the  amount  and  complexity  of  the  information  being  provided.  The  ongoing  availability  of 
the  accumulated  materials  allowed  students  to  retrieve  and  review  them  as  required.  Even  so,  it  seemed  a little 
overwhelming  at  times. 

...the  fact  that  they  added  [ideas]  as  they  learnt  them  means  you ’ve  got  to  ask  ‘Well  where  does  that  fit 
into  this?’  So  that  forces  them  to  make  those  links.  (Teacher  interview  - Murder  Under  the  Microscope) 

It  is  in  this  way  that  online  projects  also  differ  significantly  from  other  online  activities.  A partnership 
developed  between  the  online  component  of  the  project  and  the  necessary  activities  that  took  place  in  other 
spheres  of  the  class’  work  - both  on  and  off  computer.  Student  activities  take  centre  stage;  the  construction  of 
ideas  takes  place  between  students.  The  technologies  themselves  (the  computer  and  the  network)  recede  into  the 
background  of  complex  learning  tasks,  perhaps  more  so  than  in  other  styles  of  online  activity. 

The  Murder  project  provided  a wealth  of  support  materials:  appropriately  pitched  reference  materials, 
formats  for  organising  and  presenting  information  and  lists  of  web  resources  for  students;  planning  and 
scheduling  advice,  suggested  sequences  of  lessons,  assessment  formats  for  teachers.  Support  materials  are 
intended  to  assist  where  and  when  needed.  They  are  neither  prescriptive  nor  exhaustive.  They  do  however, 
provide  much  needed  supports  for  teachers,  elaborating  possible  implementation  strategies  and  ways  to  manage 
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the  knowledge  building  processes. 

While  the  online  interaction  was  the  dominant  function  of  the  ME  simulation,  the  final  face-to-face 
meeting  was  highly  valued  by  all  involved.  Again,  the  balance  of  virtual  and  real  interaction  was  a major  benefit 
of  the  experience.  The  depth  of  learning  that  occurred  during  the  three  weeks  of  online  interaction  prepared 
students  for  the  often  confronting  task  of  arguing  the  points  face-to-face. 

/ learnt  it  so  well.  I found  when  I finished  - 1 don ’t  know  why  that  stuck. . . . you  ’ve  had  all  this  lead  up  to  it . 

You  actually  believe  what  you  are  saying.  ( Student  interview,  Middle  East  simulation) 

It  is  certainly  conceivable  to  suggest  that  the  activity  could  have  taken  place  entirely  in  a face-to-face  classroom 
environment.  However,  being  online  added  several  significant  dimensions.  Shifting  the  major  activity  away 
from  a face-to-face  interaction  increased  students’  ability  to  construct  arguments  in  considered  ways,  working 
collaboratively  to  explore  ideas,  plan  actions  and  respond  to  the  initiatives  of  other  characters,  without  being 
interrupted  (Wills  & Ip,  2002).  An  immediate  comparison  was  possible. 

...it  let  everyone  get  a say.  In  the  conference  you  don ’t  have  time  to  search  for  that  perfect  word  that 

would  just  fit  in  - it’s  hard  to  be  articulate  - in  e-mails  you  can  think  about  it...  (Student  interview,  Middle 

East  simulation ) 

Events  unfolded  on  a daily  basis,  sustaining  the  momentum  while  maintaining  the  depth  of  responses.  As  for 
Murder,  the  online  environment  contributed  to  the  pace  of  the  action  and  interaction. 

Working  electronically,  interactions  between  group  members  did  not  have  to  occur  synchronously.  The 
workload  was  frequently  divided,  with  group  members  taking  the  load  on  different  days,  while  maintaining 
shared  responsibility  through  systems  of  individual  drafting  and  group  review,  amendment  or  endorsement.  The 
simulation  environment  provided  a ‘diary’  area  where  character  group  members  could  privately  communicate 
with  each  other.  Not  all  groups  made  use  of  this,  preferring  to  use  instant  messaging  or  the  telephone,  or  even 
discussing  and  planning  at  school.  The  intensity  of  the  project  often  led  to  combinations  of  these  being  used 
simultaneously.  The  project  made  it  imperative  that  students  plan  and  manage  their  participation,  at  the  same 
time  as  providing  support  structures  to  do  so. 

In  the  ME  simulation,  involvement  of  another  school  was  only  made  possible  by  working  online. 
Students  commented  that  this  enhanced  the  realness  of  the  situation,  adding  new  perspectives  and  a greater 
range  of  ideas  and  unpredictable  responses.  As  they  did  not  know  the  others,  they  communicated  entirely  in 
role.  Wills  and  Ip,  (2002)  suggest  such  anonymity  makes  participation  more  comfortable,  especially  for  adult 
learners.  For  the  school  students  being  online  also  meant  that  existing  relationships  were  minimised;  the  action 
was  ‘unable  to  be  influenced’  by  existing  friendships.  Most  importantly  it  added  to  the  authenticity  of  the 
action;  communication  occurred  between  the  ‘characters’  rather  than  friends  and  classmates. 

It  didn’t  feel  like  just  talking  to  kids ! 


Conclusions 

There  is  no  revolution  happening  here  and  perhaps  we  should  stop  expecting  one.  I have  learnt,  as  have 
the  teachers  in  my  study,  that  online  projects  have  a deal  to  offer  in  creating  authentic,  problem-based  learning 
experiences  for  students  and  in  making  effective  use  of  online  technologies,  although  not  without  sizeable 
concerns  to  be  addressed. 

Students  were  asked  if  they  would  recommend  the  project  to  others.  An  overwhelmingly  positive 
response  was  tempered  by  similar  issues  identified  by  students  in  both  projects:  the  amount  of  time  it  took  away 
from  other  set  tasks  and  the  difficulty  found  by  some  in  working  in  their  allocated  groups. 

More  experienced  computer  users  in  both  age  groups  were  most  vocally  critical  of  any  online  aspects 
that  didn’t  measure  up  to  their  (outside  school)  experiences:  the  relatively  limited  ‘flashiness’  of  the  Murder 
graphics,  ‘cripplingly’  slow  speed  of  the  simulation  chat  facility.  This  sets  immediate  challenges  for  schools,  ‘to 
be  in  the  game’  both  in  terms  of  quality  of  functionality  and  interface  design  that  supports  the  purposes  of  the 
sites.  This  is  not  easy.  We  know  that  graphics,  functions  and  interface  design  of  recreational  software  are  the 
result  of  a multi-billion  dollar,  cut-throat  industry  built  on  rapid  updating  and  expanding  repertoires  of  effects 
and  features.  It  is  impossible  for  the  resource-poor  education  sector  to  keep  up  technically.  Students’  further 
comments  were  somewhat  reassuring.  They  suggested  that  ‘bells  and  whistles’  are  not  critical,  but  reliable. 
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efficient  function  is,  with  a style  of  presentation  that  supports  the  purpose  without  attempting  to  be  more  than  it 
is.  The  greatest  criticism  was  of  ‘try  hard’  failures. 

The  teachers  would  all  do  it  again,  too...  “but  not  all  the  time!”  They  acknowledged  that  the  projects 
created  a style  of  learning  that  is  rewarding  and  adds  tangible  benefits  to  children:  realism,  purpose,  authentic 
process  and  valuable  content,  emphasis  on  student  knowledge  building.  But  it  requires  a balance.  Projects  are 
time  consuming  (if  not  all-consuming)  and  have  more  than  just  the  potential  to  take  time  away  from  other, 
equally  important  learning  activities  of  the  class. 

Ways  of  resolving  the  tension  between  the  time  taken  and  the  value  gained,  requires  further 
investigation.  Can  we  accept  that  the  time  it  takes,  is  the  time  it  takes  and  therefore  is  worth  it?  Or  should  the 
projects  be  scaled  back  so  they  more  manageably  become  an  ‘ordinary’  part  of  class  activities.  At  what  point  in 
scaling  is  the  value  lost?  Is  it  sufficient  to  implement  projects  only  periodically,  particularly  if  the  models  of 
changed  pedagogy  can  be  incorporated  in  teachers’  design  of  other  learning  activities? 

Teachers  certainly  did  not  want  to  have  to  create  projects  themselves.  They  have  neither  the  time  or 
expertise.  Definite  value  lay  in  the  projects  being  available  for  teachers  to  participate  in,  as  convenient. 

The  continued  existence  (individually  and  collectively)  of  online  projects  relies  on  teachers  and 
students  finding  the  experiences  useful,  both  in  delivering  learning  benefits  and  providing  support  for  changed 
pedagogy.  The  evidence  from  these  two  cases  suggests  that,  with  support,  this  can  be  achieved. 

The  study  has  significant  implications  for  education  systems  in  the  design  and  implementation  of 
online  projects  as  part  an  effective  online  learning  provision  for  schools.  While  they  are  enthusiastically 
implemented  by  teachers  who  value  the  student  learning  achieved  and  the  support  provided  within  the  project 
environments,  wider  implementation  remains  sporadic,  at  best.  Comple  x problem-solving  projects  require  time 
and  expertise  to  develop  and  maintain,  far  beyond  the  capacity  of  individuals  or  even  groups  of  teachers  to 
sustain.  Both  these  issues  suggest  the  increased  need  for  systemic  development  and  support  for  projects, 
particularly  in  partnership  with  other  organisations.  Yet  in  NSW  and  most  other  states  of  Australia,  they  remain 
the  ‘poor  relation’  of  online  activities:  underfunded  and  outside  priority  e-learning  development  areas. 

In  this  highly  conflicted  area  of  investment  in  ICT  and  the  search  for  purposeful  learning  uses  of  the 
Internet,  online  projects  present  a teaching  and  learning  approach  that  can  deliver  on  some  of  the  much- 
acclaimed  potential  - primarily  because  they  promote  changes  in  practice  that  are  concerned  with  much  more 
than  just  the  technology. 

'The  Middle  East  simulation,  described  later  in  this  paper,  was  conducted  predominantly  in  the  virtual  project 
environment.  Student  activity  took  place  outside  class  time  and  locations  included  students’  homes,  the  school 
library  and  other  venues  where  Internet  access  was  available.  In  place  of  direct  observations,  descriptions  of  the 
learning  activity  and  students’  perceptions  were  obtained  through  interviews  with  six  groups  of  students  (two 
from  each  of  the  three  participating  classes)  at  three  points  during  the  project,  and  diaries  of  participation 
completed  by  four  volunteers,  two  from  each  school. 

2The  NSW  primary  curriculum  brings  science  and  technology  subjects  together  in  a single  syllabus,  Science  & 
Technology  K-6.  The  three  learning  processes  of  Investigating,  Designing  and  making  and  Using  technology 
underpin  all  learning  in  the  area. 
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Past  Technologies,  Practice  and  Applications:  A Discussion  on  How  the 
Major  Developments  in  Instructional  Technology  in  the  20th  Century 
Affect  the  Following  Qualities  - Access,  Efficiency,  Effectiveness,  and 

Humaneness 

Khe  Foon  Hew 
Indiana  University 

Introduction 

The  Association  for  Educational  Communications  and  Technology  defines  Instructional  Technology  as 
the  “theory  and  practice  of  design,  development,  utilization,  management,  and  evaluation  of  processes  and 
resources  for  learning’'  (Seels  & Richley,  1994).  From  the  above  definition,  it  can  be  seen  that  Instructional 
Technology  can  be  considered  in  terms  of  the  use  of  media  (or  resources)  and  the  use  of  “systematic 
instructional  design  procedures’’  (Reiser,  2002,  p.  28)  that  includes  the  processes  such  as  design,  development, 
utilization,  management,  and  evaluation.  For  the  purpose  of  this  article,  I shall  refer  the  latter  to  instructional 
design  for  instructional  purposes.  I summarize  the  history  of  Instructional  Technology  in  the  20lh  century  using 
these  two  qualities  - media  and  instructional  design.  Also  included  in  this  article  is  a discussion  on  how  these 
two  qualities  affect  the  following  four  qualities  of  instructional  technology:  access,  efficiency,  effectiveness, 
and  humaneness. 


History  of  Media  Developments  in  Instructional  Technology 

According  to  Clark  (1996),  there  are  five  possible  perspectives  on  media:  (a)  media  as  machines,  (b) 
media  as  tutors,  (c)  media  as  socializing  agents,  (d)  media  as  motivators  for  learning,  and  (e)  media  as  mental 
tools  for  thinking  and  problem  solving.  For  the  purpose  of  this  article,  the  first  perspective  is  taken.  Drawing 
upon  Dale’s  (1969)  and  Reiser’s  (2002)  work,  I have  classified  the  history  of  media  in  Instructional  Technology 
into  the  following  six  main  periods  - Silent  visual  media,  Audiovisual  materials,  Instructional  television. 
Personal  computers,  Internet,  and  wireless  tools. 

Silent  Visual  Media 

Media  developments  in  instructional  technology  essentially  began  with  silent  visual  media  that  were 
housed  in  school  museums  (Saettler,  1990).  These  silent  media  includes  stereographs,  slides,  study  prints, 
charts,  and  photographs  (Saettler,  1968;  Dale,  1969).  With  the  advent  of  the  motion  picture  projector,  silent 
visual  instructional  films  soon  appeared  in  the  educational  landscape.  In  1910,  the  public  school  system  of 
Rochester,  NY,  became  the  first  to  adopt  silent  instructional  films  for  instructional  purposes  (Reiser,  2002). 

That  same  year  also  saw  the  publication  of  the  first  catalog  of  instructional  films  (Reiser,  2001). 

Audiovisual  Materials 

With  the  advent  of  media  incorporating  sound,  the  silent  visual  era  soon  gave  way  to  what  is  known  as 
the  audiovisual  instruction  movement.  As  a result,  sound  incorporated  motion  pictures  became  a reality,  which 
led  to  increased  interest  in  the  use  of  media  to  enhance  learning.  Dale  (1969),  for  example,  summarized  twelve 
benefits  of  motion  pictures,  which  include  the  ability  to  create  reality  or  reveal  the  invisibility,  compel  attention, 
and  promote  an  understanding  of  abstract  relationships. 

In  the  1920s,  the  radio  was  invented,  and  soon  it  was  used  in  carrying  on  various  types  of  educational 
activities.  Some  of  its  characteristics  that  made  it  educationally  valuable  were  its  low  cost,  its  ability  to  bring 
dramatic  feeling  into  the  classroom,  and  the  fact  that  listening  can  foster  imagination  on  the  part  of  the  listener 
(Dale,  1969). 

Instructional  Television 

The  advent  of  television  soon  followed  in  the  1950s.  There  was  much  interest  in  the  use  of 
instructional  television  then,  so  much  so  that  by  1955,  there  were  17  instructional  television  channels  in  the 
U.S.,  and  by  1960,  had  increased  to  more  than  50  (Blakey,  1979).  Dis  criminating  between  the  terms 
“educational  television”  and  “instructional  television”,  Dale  highlighted  some  major  characteristics  of  the  latter: 
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(a)  instructor-guided,  (b)  systematic  with  objectives  of  the  course  and  planned  learning  experiences,  (c)  ordered 
and  sequential,  and  (d)  integrated  to  other  learning  experiences  such  as  practice,  reading,  laboratory,  and 
writing.  Central  to  the  great  interest  in  the  use  of  instructional  television  was  the  belief  that  this  particular 
medium  has  the  potential  to  bring  demonstrations  to  the  classroom,  handles  films  and  other  changes  from 
straight  classroom  presentation  with  a minimum  of  transitional  difficulty,  concentrates  attention,  and  provides  a 
change  of  pace,  often  a lift,  for  the  classroom  (Schramm  cited  in  Dale,  1969,  p.  356). 

The  interest,  however,  did  not  prevail  long  and  by  mid-1960s  it  had  abated  (Reiser,  2001)  due  to 
reasons  such  as  the  inability  of  television  alone  to  adequately  present  the  various  conditions  necessary  for 
student  learning  (Tyler,  1975). 

Personal  Computers 

The  next  media  to  grab  the  attention  of  educators  was  the  personal  computer  in  the  1980s.  The  use  of 
personal  computers  for  instructional  purposes  has  since  increased  and  by  2003,  virtually  all  U.S.  schools  have 
some  personal  computers  and  the  most  recent  ratio  of  students  to  computers  was  fewer  than  four  to  one  (Market 
Data  Retrieval,  2003). 

Internet 

The  explosion  of  the  Internet  and  world-wide  web  (WWW)  soon  followed  in  the  wake  of  the  advent  of 
personal  computers.  In  1991,  Tim  Berners-Lee  completed  the  original  software  for  the  WWW,  the  hypertext 
system  he  had  first  proposed  in  1989.  He  had  envisioned  the  WWW  as  a shared  information  space,  a web  of 
hypertext  documents,  within  which  people  can  communicate  with  each  other  and  with  corrputers  (Moschovitis, 
Poole,  Schuyler,  & Senft,  1999).  The  use  of  the  Internet  and  WWW  is  now  ubiquitous,  with  98%  of  U.S  public 
schools  already  connected  to  the  Internet  in  Fall  2001  (Cattagni  & Farris,  2001).  The  student-per-Internet- 
connected  computer  ratio  was  now  4.3:1  (Education  Week,  cited  in  Molenda  & Bichelmeyer,  in  press). 

Wireless  Tools 

Currently,  the  use  of  wireless  tools  such  as  pocket  PCs  or  personal  digital  assistants  (PDA)  is 
increasingly  widespread  within  K-12  and  higher  education.  According  to  Park  and  Staresina  (2004),  about  8% 
of  schools  nationwide  provide  PDAs  for  their  teachers  and  4%  provide  them  for  their  students.  For  teachers,  the 
PDAs  are  useful  for  lesson  preparation  and  classroom  management,  such  as  taking  attendance.  For  students,  the 
PDAs  are  used  as  digital  readers  and  graphing  calculators,  for  word  processing,  and  other  specific  instructional 
activities  such  as  concept  mapping  (ISTE,  cited  in  Molenda  & Bichelmeyer,  in  press). 

Effects  of  the  Major  Media  Developments  on  Access,  Efficiency,  Effectiveness  and 

Humaneness 

One  very  interesting  recurrent  pattern  can  be  seen  throughout  the  development  of  media  for 
instructional  purposes,  which  is  the  comparison  between  the  anticipated  and  real  effect  of  media  on  instructional 
practices  (Reiser,  2001).  From  Thomas  Edison’s  1913  famous  prediction  that  films  will  replace  every  other 
media,  or  the  advent  of  computer  instructional  programs  to  the  hype  created  every  time  a new  instructional 
media  is  discovered,  expectations  have  been  usually  greater  than  outcomes.  Many  of  the  media  research  studies 
show  that  students  learn  equally  and  effectively  well  regardless  of  the  types  of  media  used  (Clark,  1994; 

Russell,  1999),  leading  some  researchers  to  believe  that  it  is  the  instructional  methods  rather  than  media  that  can 
influence  the  effectiveness  of  student  learning  (Clark,  1994). 

It  seems  then  that  the  effects  of  media  per  se  on  instructional  technology  are  geared  more  towards  the 
access,  efficiency,  and  humaneness  rather  than  the  effectiveness  aspect.  Media  helps  increase  access  to  learning 
by  giving  students  opportunities  that  never  existed  before.  The  use  of  the  Internet  and  teleconferencing,  for 
example,  enables  students  to  sign  up  for  courses  from  virtually  all  sorts  of  geographical  locations.  Students  from 
far-flung,  hard-to-reach  places  can  now  have  easy  access  to  learning.  So  do  students  who  have  physical 
disabilities  that  rendered  them  immobile. 

Closely  related  to  the  idea  of  access  is  efficiency,  which  can  include  concepts  such  as  cost  efficiency, 
time  efficiency,  energy  efficiency,  and  “delivery -of-information”  efficiency.  The  use  of  the  Internet  is  a good 
example  of  “delivery -of-information”  efficiency  because  information  can  now  be  easily  structured  and 
presented  to  many  learners  at  one  time.  It  is  also  cost  efficient  to  design  and  develop  courses  over  the  Internet 
because  as  the  number  of  students  taking  an  online  course  increased,  the  development  costs  were  spread  across 
a large  student  body,  making  the  development  cost  per  student  low  (Shearer,  2003).  Meanwhile,  the  use  of 
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management  tools  such  as  WebCT,  and  BlackBoard  allows  instructors  a greater  degree  of  flexibility  and  ease  in 
terms  of  updating  and  revising  courses  (Shearer,  2003).  This  has  brought  about  time  and  energy  efficiency  for 
the  instructors. 

The  development  of  certain  media  has  also  improved  the  humaneness  aspect  by  providing  rich  sensory 
experiences  and  individualized  learning  to  students.  Television  and  computers,  for  example,  provide  students 
with  vivid  three-dimensional  images  that  may  help  them  remember  information  better,  than  printed  materials 
can. 


Major  Instructional  Design  Developments  in  Instructional  Technology 

Besides  the  major  media  developments,  there  were  also  major  instructional  design  developments  that 
shaped  the  field  of  instructional  technology.  Many  authors  have  classified  these  instructional  design 
developments  into  various  categories;  these  have  typically  been  done  in  more  or  less  a chronological  fashion 
(e.g.  Reiser,  2001,  2002).  In  this  article,  however,  I chose  to  succinctly  divide  the  periods  of  instructional  design 
developments  in  terms  of  its  design  paradigms,  ontological,  and  epistemological  perspectives  drawing  mainly 
upon  the  work  of  Wilson  (1997,  2004),  Driscoll  (2000),  and  Mertens  (1998);  yielding  the  following  three 
categories  - (a)  behaviorist  paradigm,  (b)  conditions-of-learning  paradigm,  and  (c)  constructivis  t paradigm.  I 
decided  to  do  this  because  I felt  that  such  a move  would  better  capture  the  real  essence  of  instructional  design 
developments  compared  to  a chronological  method  per  se.  Table  1 shows  these  paradigms  as  well  as  the 
ontological  and  epistemological  perspectives  associated  with  them. 


Design  paradigm 

Ontological  perspective 

Epistemological  perspective 

Behaviorist  paradigm 

Positivism 

Objectivism 

Conditions-of-learning  paradigm 

Positivism 

Objectivism 

Constructivism 

Interpretivism  and  sometimes 

Contextualism/situated 

Postmodernism 

knowledge 

Table  1.  Design  paradigm,  ontological  and  epistemological  perspectives 


Behaviorist  paradigm 

One  of  the  major  concepts  that  appeared  in  the  1910s  was  the  claim  by  John  Watson  that  human  and 
animal  behaviors  were  the  only  legitimate  areas  of  study  for  psychologists.  This  signaled  the  birth  of 
behaviorism  as  an  approach  to  psychology  and  learning  that  emphasizes  observable  and  measurable  behavior 
but  ignores  mental  processes.  Watson’s  idea  was  in  essence  a “descriptive  S-R  psychology  whose  goal  was  to 
predict  and/or  control  behavior”  (Saettler,  1990,  p.  287).  Edward  Thorndike  soon  followed  with  his  theory  of 
connectionism,  which  is  a descriptive  learning  theory  made  up  of  three  laws  - the  law  of  effect,  the  law  of 
exercise,  and  the  law  of  readiness.  These  laws  are  founded  on  the  stimulus-response  notion  - “a  neutral  bond 
would  be  established  between  the  stimulus  and  the  response  when  a particular  stimulus  produced  a satisfying 
response  with  a given  environment”  (Saettler,  1990,  p.  55).  Learning  takes  place  when  these  bonds  formed  into 
patterns  of  behavior.  Building  on  Thorndike’s  and  Watson’s  ideas,  B.  F.  Skinner  came  up  with  the  theory  of 
operant  conditioning,  which  basically  states  that  “if  the  occurrence  of  an  operant  is  followed  by  presentation  of 
a reinforcing  stimulus,  the  strength  probability  is  increased”  (Saettler,  1990,  p.  7 1).  This  led  to  a science  of 
instruction  with  systematic  methods  of  modifying  behaviors,  and  determining  the  types  of  response  patterns 
associated  with  different  reinforcement  schedules  (Ferster  & Skinner,  1957). 

Central  to  the  behaviorist  paradigm  is  the  notion  of  using  experimental  approaches  in  the  study  of 
learning.  The  works  of  Hermann  Ebbinghaus  Edward  Thorndike,  and  Ivan  Pavlov,  for  instance,  bore  testimony 
to  it.  This  notion  of  using  experimental  approaches  as  a method  of  study  resonates  with  the  positivist 
ontological  perspective.  According  to  Mertens  (1998),  the  underlying  assumptions  of  positivism  include  the 
belief  that  the  social  world  can  be  analyzed  in  the  same  way  as  the  natural  way,  that  there  is  a technique  for 
studying  the  social  world  which  is  value-free,  and  that  explanations  of  a causal  nature  can  be  given.  In  positivist 
thinking,  objectivity  is  important,  whereby  the  researcher  simply  manipulates  and  observes  in  a dispassionate 
manner. 

Conditions-of-learning  paradigm 

B.  F.  Skinner’s  behaviorism  work  on  contingencies  of  reinforcement  and  programmed  instruction  was 
the  antecedent  to  the  next  paradigm-  the  conditions-of-learning.  As  noted  clearly  by  Wilson  (2004): 

In  the  days  of  programmed  instruction,  researchers  held  to  a few  general  principle  s of 
learning,  based  on  behavioral  psychology,  which  were  thought  to  apply  universally  to  all  settings  and 
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organisms.  Results  of  programmed  instruction,  however,  showed  that  some  strategies  worked  better 
than  others,  depending  on  conditions.  This  led  Lumsdaine  (cited  in  Wilson,  2004)  and  others  to 
articulate  a vision  for  an  emerging  science  of  instruction:  through  factorial  experiments,  instructional 
scientists  would  develop  a sophisticated  series  of  rules,  sub-rules,  and  meta-rules  for  employing 
instructional  strategies  to  teach  different  kinds  of  content  in  different  settings. 

The  belief  that  certain  strategies  would  be  more  appropriate  for  certain  context  led  to  the  notion  of  the 
conditions-of-learning  paradigm.  In  essence,  this  paradigm  posits  that  there  exists  a hierarchy  of  learning 
outcomes,  and  for  each  desired  outcome,  there  exists  a set  of  conditions  which  leads  to  learning  (Wilson,  1997). 
Some  of  the  noted  authors  and  thinkers  of  this  paradigm  include  Benjamin  Bloom  (who  developed  the 
taxonomy  of  educational  objectives  for  the  cognitive  domain),  Robert  Gagne  (who  introduced  five  domains  of 
learning  and  the  conditions  necessary  for  each  of  them,  as  well  as  the  nine  events  of  instruction  to  promote  the 
attainment  of  any  type  of  learning  outcome),  and  Charles  Reigeluth  (who  discussed  the  various  conditions  in  his 
Green  books). 

In  addition  to  the  belief  of  a rule  set  that  links  conditions,  instructional  methods,  and  learning 
outcomes,  there  were  two  other  important  ideas  that  emerged  during  this  paradigm:  (a)  the  systems  approach  in 
instructional  design,  and  (b)  the  cognitive  information-processing  theory.  In  the  systems  approach,  the  design  of 
instruction  is  divided  into  small  manageable  components  or  procedures,  where  the  output  of  one  procedure 
becomes  the  input  of  the  next  one.  A good  example  would  be  the  ADDIE  systems  approach  which  is  made  up 
of  five  procedures:  Analysis,  Design,  Development,  Implementation,  and  Evaluation.  The  design  of  instruction 
proceeds  more  or  less  in  a linear  fashion,  beginning  with  the  Analysis  phase,  and  ending  with  Evaluation.  The 
burgeoning  interest  in  the  systems  approach  spawned  many  instructional  design  models  (e.g.  the  Dick  & 
Carey’s,  1978),  as  well  as  methodologies  and  concepts  (e.g.  Mager’s  1962  behavioral  objectives,  Glaser’s  1963 
criterion-referenced  testing,  and  Scriven’s  1967  formative  evaluation). 

The  second  important  emerging  idea  in  this  paradigm  was  the  cognitive  information-processing  theory 
which  posits  that  information  is  transformed  as  it  passes  through  three  main  stages  of  memory:  sensory 
memory,  short-term  memory,  and  long-term  memory  (Atkinson  & Shiffrin,  1968).  As  this  idea  gained 
acceptance  and  popularity,  many  researchers  began  to  incorporate  it  in  their  studies.  One  of  the  most  notable 
works  based  on  the  cognitive  information-processing  theory  is  the  nine  events  of  instruction,  developed  by 
Robert  Gagne. 

The  conditions-of-learning  paradigm  also  appeared  to  resonate  with  the  positivist  ontological 
perspective.  This  is  because  many  researchers  in  this  period  still  used  the  experimental  design  as  the  main 
research  tool  in  trying  to  isolate  instructional  methods  most  appropriate  for  certain  contexts.  As  noted  by 
Driscoll  (2000),  both  Skinnerian  Behaviorism,  and  Gagne’s  instructional  theory  rest  on  objectivist  assumptions. 

Constructivist  paradigm 

Constructivism,  which  states  that  learning  is  a process  of  knowledge  construction,  rather  than 
acquisition,  began  to  rise  in  prominence  beginning  in  the  early  90s.  Duffy  and  Cunningham  (1996)  postulated 
that  some  of  the  assumptions  that  are  adopted  by  constructivists  include  the  following:  (1)  all  knowledge  is 
constructed  (i.e.,  learners  construct  understanding  of  the  world  for  themselves  (Winn,  2003)),  (2)  many  world 
views  can  be  constructed,  hence  there  will  be  multiple  perspectives,  (3)  knowledge  is  context  dependent,  thus 
learning  should  occur  in  contexts  to  which  it  is  relevant,  (4)  learning  is  mediated  by  tools  and  signs,  (5)  learning 
is  an  inherently  social-dialogical  activity,  (6)  learners  are  distributed,  multidimensional  participants  in  a socio- 
cultural process,  and  (7)  knowing  how  we  know  (reflexivity)  is  the  ultimate  human  accomplishment. 

In  contrast  to  the  previous  conditions-of-learning  paradigm  which  is  based  on  a reductionist  view  (e.g. 
the  use  of  task  analysis  to  decompose  a subject  into  various  sub-subjects  according  to  its  learning  outcomes; 
then  instruction  is  ordered  from  simple  to  complex),  the  constructivist  paradigm  celebrates  complex,  authentic 
tasks  that  are  not  broken  into  smaller  components.  It  emphasizes  the  creation  of  rich  learning  environments  and 
the  use  of  scaffolding  to  help  learners  gain  the  knowledge  and  skills  of  a practitioner. 

The  constructivist  paradigm  is  most  aligned  with  the  interpretive  ontological  perspective.  According  to 
Mertens  (1998),  the  basic  assumptions  guiding  the  interpretive  ontological  position  are  that  knowledge  is 
socially  constructed  by  people,  and  that  perceptions  of  reality  may  change  throughout  the  process  of  study.  It 
rejects  the  notion  of  objectivism  Constructivism  is  also  consistent  with  the  postmodern  perspective.  Although  I 
agree  with  Wilson’s  (1997)  observation  that  not  all  constructivists  are  postmodern  in  their  orientation,  I would, 
however,  argue  that  most  of  the  concepts  of  constructivism  appear  to  be  founded  on  the  key  ideas  of 
postmodernism.  For  example,  postmodernism  posits  that  knowledge  is  constructed  by  people,  reality  is 
multiperperspectival,  and  thinking  is  an  interpretive  act  (Wilson,  1997).  These  key  ideas  are  congruent  with 
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those  held  by  constructivists. 

Effects  of  the  Major  Instructional  Design  Paradigms  on  Access,  Efficiency, 

Effectiveness  and  Humaneness 

I believe  the  developments  of  the  major  instructional  design  paradigms  primarily  influenced  the 
effectiveness  and  humaneness  aspects  of  instructional  technology.  For  example,  design  concepts  based  on  the 
behaviorist,  conditions-of-learning,and  constructivism  paradigms  all  strive  to  improve  the  effectiveness  of 
instruction,  with  each  camp  advocating  its  own  advantages,  sometimes  to  unproductive  arguments.  Nonetheless, 
there  has  been  a consensus  in  recent  years  among  educators  that  the  design  concepts  based  on  different 
paradigm  each  has  its  own  place  and  value  best  effective  for  certain  types  of  learning  outcomes  such  as: 

For  discrimination  (recalling  facts),  generalizations  (defining  and  illustrating  concepts),  associations 
(applying  explanations),  and  chaining  (automatically  performing  a specified  procedure)  types  of  learning 
outcome,  design  concepts  based  on  the  behaviorist  paradigm  would  be  a good  choice  (Ertmer  & Newby,  1993). 

For  advanced  knowledge  acquisition  and  problem-solving  in  ill-structured  domains  types  of  learning, 
design  concepts  based  on  the  constructivist  paradigm  would  work  best  (Jonassen,  1991). 

However,  in  terms  of  humane  benefits,  there  are  certain  design  concepts  that  lend  themselves  better 
than  others  in  providing  rich  authentic  experiences.  Design  concepts  associated  with  constructivist  paradigm 
which  require  learners  to  solve  realistic  problems,  and  take  ownership  of  the  learning  process  (Driscoll,  2000), 
is  a good  example . 


Summary 

There  have  been  many  developments  both  in  the  media  and  design  process  aspects  of  instructional 
technology  in  the  20th  century.  These  have  in  turn  affected  each  of  the  four  instructional  technology  qualities: 
access,  efficiency,  effectiveness,  and  humaneness.  The  developments  of  film,  radio,  television,  and  computers 
in  the  have  each  increased  the  access,  efficiency,  and  humaneness  aspects.  I believe  that  media  per  se,  however, 
do  not  directly  affect  the  effectiveness  of  learning  because  it  is  the  method  or  process  that  determines  it.  De  sign 
concepts  such  as  behaviorist  paradigm,  conditions  of  learning,  as  well  as  constructivism,  have  also  affected  the 
effectiveness  and  humaneness  of  instructional  technology  through  the  provision  of  rich  experiences  and 
individualized  learning  for  students. 
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Abstract 

Students  taking  hybrid  or  online  classes  are  often  unprepared  for  the  kinds  of  skills  that  are  needed  to 
be  successful  in  this  environment.  This  report  provides  an  overview  of  one  approach,  an  interactive  CD-ROM 
(SPARK),  that  faculty  can  use  to  assist  students  in  narrowing  the  gap  between  needed  online  learning  skills  and 
their  current  technical  knowledge. 


Igniting  the  SPARK: 

Supporting  the  Technology  Needs  of  Online  Learners 

The  popularity  of  online  learning  continues  to  transform  the  educational  landscape.  As  more  faculty 
redesign  courses  to  meet  the  demands  of  education  in  the  21st  Century,  some  students  can  be  left  behind. 
Students  who  have  not  used  information  technology  in  previous  school  experiences  and  those  who  are  returning 
to  school  after  a long  hiatus  from  higher  education  are  of  particular  concern.  Even  those  students  who  consider 
themselves  to  be  technically  proficient  may  have  developed  bad  habits  over  the  years  that  create  barriers  for 
them  in  the  online  context . Faculty  should  recognize  this  potential  “digital  divide”  and  assure  their  students 
have  the  tools  they  need  to  be  s uccessful  in  online  learning  experiences. 

Online  courses  suffer  from  high  attrition  rates.  A possible  explanation  is  that  students  are  not 
adequately  prepared.  According  to  Rowntree  (1995),  one  of  the  key  skills  areas  that  students  identify  as 
requiring  a “steep  learning  curve”  for  online  learning  includes  computing  skills  (p.  212).  The  Student 
Preparation  and  Resource  Kit  (SPARK)  was  created  to  address  gaps  in  knowledge  between  needed  online 
learning  skills  and  students’  knowledge  deficits.  SPARK  has  been  piloted  with  two  groups  of  nursing  students: 
19  undergraduates  and  18  graduate  students.  Following  is  a description  of  SPARK,  related  definitions,  a brief 
review  of  usability  literature  and  a report  of  student  evaluations  of  the  CD-ROM. 

Description  of  SPARK 

SPARK  was  created  through  a partnership  of  the  College  of  Nursing  and  MediaKube,  LLC,  a digital 
solutions  provider  and  funded  by  the  Arizona  Regents  University.  The  CD-ROM  was  planned  to  be  easy  to 
navigate,  entertaining,  and  conversational.  The  decision  to  use  this  instructional  style  had  two  positive 
implications.  First,  students  who  considered  themselves  computer  novices  would  be  more  likely  to  retain 
information  presented  in  a non-threatening  manner.  Second,  students  who  felt  they  already  were  familiar  with 
the  material  would  be  enticed  to  explore  the  content  for  the  entertainment  value. 

A significant  challenge  was  that  the  program  had  to  effectively  present  items  of  a technical  nature  in  a 
way  that  was  not  daunting  for  the  user.  Wherever  possible,  real-world  analogies  were  used  to  relate  terminology 
to  something  with  which  the  student  was  likely  to  be  familiar.  For  example,  a flatbed  scanner  is  compared  to  a 
traditional  copy  machine  with  the  noted  exception  that  the  scanner  output  is  sent  to  a computer  via  a digital 
signal  instead  of  printed  onto  a piece  of  paper.  Humor  was  injected  throughout  to  make  the  content  less 
intimidating  and  to  facilitate  the  description  of  complex  subjects.  Remediation  for  wrong  answers  was  provided 
in  a helpful  and  friendly  manner.  The  scripting  allows  students  to  repeat  a question  just  to  find  out  how  the 
software  reacts  to  the  wrong  answers.  Learning  why  an  answer  is  wrong  can  often  be  more  educational  than 
simply  knowing  the  correct  response. 

SPARK  is  an  appealing  visual  experience  with  plenty  of  motion  and  imagery.  This  delivery  style  helps 
direct  the  immediate  attention  of  the  student,  while  at  the  same  time  giving  them  a mental  image  to  recall  at  a 
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later  date  when  they  need  to  apply  the  information  they  have  learned.  Where  appropriate,  animated  simulations 
demonstrate  the  appropriate  steps  in  a particular  task  prior  to  requiring  the  user  to  performthe  task. 

For  ease  of  use,  SPARK  is  configured  to  launch  automatically  when  the  CD  is  inserted  into  a PC.  The 
navigation  in  SPARK  is  designed  to  be  as  unobtrusive  as  possible,  while  still  providing  a substantial  degree  of 
control  for  the  student.  The  replay  and  skip  buttons  allow  the  student  to  quickly  maneuver  within  a topic,  while 
a click  of  the  map  button  offers  them  a hierarchical  view  of  the  entire  content  tree.  The  student  can  navigate  to 
any  other  program  topic  with  just  three  or  four  clicks. 

SPARK  Configuration  and  Navigation 

The  program  begins  with  an  animated  series  of  credits  and  title  screens.  The  voice-over  narrator  starts 
by  asking,  “Is  this  the  first  time  you’ve  sat  down  to  go  through  this  CD  or  have  we  already  met?”.  A click  of 
button  A “First  time  for  me.”  takes  the  user  through  a full  introductory  sequence,  while  clicking  button  B 
“We’ve  already  met.  ” directs  them  directly  to  the  SPARK  Topic  Map.  Similar  branching  occurs  throughout 
much  of  the  introductory  section  of  the  program  for  each  main  topic.  The  Topic  Map  displays  the  main 
categories  of  information  followed  by  a layer  of  main  topics.  Below  the  main  level  is  a set  of  sub-topics  for 
each  major  category.  The  following  table  shows  the  overall  layout  of  SPARK. 


Table  1.  SPARK  Category  and  Topic  Layout 


Categories 

Hardware 

Software 

Internet 

Skills 

Navigation  He  lp 


Main  Topics 

Introduction,  CPU,  Memory,  Storage,  Input,  Output,  Connectivity 
Introduction,  OS  Software,  Applications,  Viruses 
Networks,  LAN  vs.  WAN,  World  Wide  Web 

Keyboard  Shortcuts,  File  Formats,  Using  Adobe  Reader,  Using  a Web  Browser,  Sending 
Email,  Searching,  Downloading,  File  Management 

A detailed  explanation  of  each  navigation  button  and  feature  is  displayed  on  the  Topic  Map 
screen. 


Definitions  and  Usability  Literature 

The  following  definitions  are  provided  to  clarify  the  meaning  of  various  terms  used  in  this  study: 

1.  Multimedia  is  the  convergence  of  computers  with  motion,  sound,  graphics,  and  text  (Azarmsa,  1996,  p.  2). 

2.  Hypertext  is  the  presentation  of  information  as  a linked  network  of  nodes  which  readers  are  free  to 
navigate  in  a non-linear  fashion  (Keep,  McLaughlin,  & Parmar,  1993-2000). 

3.  Hypermedia  is  a special  case  of  hypertext  that  employs  multimedia  and  describes  linked  information 
presentations  that  contain  many  forms  of  media  (Azarmsa,  1996)  that  include  sound,  video,  and  so  on 
(Keep  etal,  1993-2000). 

4.  Hyperlinks  are  the  connections  among  units  of  information  (nodes)  in  hypermedia.  This  arrangement  can 
be  described  as  a three-dimensional  web  of  information  (Dede  & Palumbo,  1991,  pp.  2-3). 

5.  Computer  literacy  level  refers  to  the  ease  with  which  a learner  is  able  to  operate  the  system  controlling  the 
hypermedia  program.  For  example,  a person  with  a low  level  of  computer  literacy  may  need  assistance 
operating  the  mouse  or  keyboard  commands  necessary  to  navigate  within  the  program. 

Hypermedia  Usability 

The  term  “hypermedia  usability”  refers  to  the  ability  to  use  a piece  of  hypermedia  software  for  the 
intended  audience.  It  pertains  to  the  ease  with  which  a learner  can  perform  a specific  search  task  for  a particular 
piece  of  information.  “Usability  is  the  combination  of  fitness  for  purpose,  ease  of  use,  and  ease  of  learning  that 
makes  a product  effective”  (Kushner,  2003).  Usability  has  been  applied  to  ‘the  Web’  (the  Internet)  for  a number 
of  years;  however,  it  is  not  specific  to  ‘the  Web’.  “Since  the  early  1980s. ...researchers  have  been  investigating 
the  usability  and  usefulness  of  hypermedia  across  a wide  spectrum  of  domains”  (Buckingham-Shum,  1996,  pp. 
1-2). 

Two  main  factors  influence  usability:  content  and  design.  Critchfield  (1998)  asserted  that  a well- 
designed  website  appears  more  credible  regardless  of  the  information  provided.  The  usability  of  instructional 
multimedia  (hypermedia)  is  vital  for  the  success  and  satisfaction  of  its  users  because  confusion  resulting  from 
poorly  designed  programs  can  be  detrimental  to  learning  performance. 

The  process  of  assessing  and  evaluating  online  content  is  subjective  and  internal  (Krug,  2000).  Several 
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approaches  for  expert -based  evaluation  of  usability  have  been  proposed  over  the  past  few  years.  According  to 
Dimitrova,  Sharp,  and  Wilson  (2001)  there  is  little  evidence  in  the  literature  regarding  the  effectiveness  of  these 
approaches . Although  expert  evaluators  are  somewhat  successful  predicting  usability  problems,  they  still  have 
difficulties  identifying  certain  types  of  learner  problems  such  as  comprehension.  Expert  evaluations  do  not 
eliminate  the  need  for  tests  with  actual  learners.  To  that  end,  an  evaluation  by  the  end-user  was  deemed 
appropriate. 

Pilot  Study  and  Evaluation 

SPARK  was  piloted  at  Arizona  State  University’s  College  of  Nursing  in  the  fall  of  2004.  The  CD- 
ROM  containing  SPARK  was  distributed  to  nineteen  members  of  an  accelerated  RN  to  BSN  program  and 
eighteen  members  of  a graduate  level  neonatal  nursing  program.  All  participants  were  allowed  to  keep  the  CD 
for  their  future  use.  Undergraduate  participants  received  extra  credit  in  their  course;  graduate  students 
volunteered  to  complete  the  evaluation  survey.  The  students  were  shown  how  to  launch  the  CD  in  class  and 
then  asked  to  take  it  home  to  review  it  on  their  own  time.  They  returned  evaluation  data  via  a seven-item  survey 
(described  below)  the  following  week. 

Evaluation  data  were  collected  using  a six  item  survey  addressing  level  of  confidence  after  viewing  SPARK,  its 
pace,  ease  of  use,  ability  to  keep  participants’  attention,  newness  of  material,  and  its  usefulness.  Participants 
ranked  their  responses  to  each  of  these  questions  on  a five-point  Likert  scale  ranging  from  “Strongly  Disagree” 
to  “Strongly  Agree”.  A comment  area  was  provided  for  each  question.  Finally,  participants  were  asked  what 
else  should  be  included  in  SPARK  as  well  as  how  long  it  took  them  to  review  the  CD. 

Results  of  Evaluation 

An  analysis  of  the  data  was  used  to  determine  what  improvements  and  modifications  should  be  made 
to  the  program.  100%  of  students  from  the  undergraduate  class  and  51%  from  the  graduate  class  responded  to 
the  survey.  Means  were  calculated  for  responses  to  the  Likert -type  scale  items;  qualitative  data  were  analyzed 
for  themes . 

Table  2.  SPARK  Survey  Items  and  Comparison  of  Means  between  Undergraduate  and  Graduate  Students 


Item 

SPARK  was  easy  to  use. 

The  topics  covered  in  SPARK  were  new  to  me. 

The  topics  covered  in  SPARK  were  useful  to  me. 

How  (narration,  self-paced  units)  topics  were  covered  in 
SPARK  kept  my  attention. 

The  pace  in  which  topics  were  covered  in  SPARK  was  just 
right. 

I feel  more  confident  about  my  computer  skills  after  using 
SPARK. 

How  much  time  did  it  take  for  you  to  review  the  materials  of 
interest  to  you?  (time  in  minutes). 

Note.  Undergraduate  (n  = 19),  Graduate  («  = 18) 

Response  scale  (1  = Strongly  Disagree,  5 = Strongly  Agree) 

Comments  were  analyzed  for  further  insights  into  participants’  experience  with  SPARK.  However, 
comments  tended  to  mirror  each  groups’  rating  of  the  evaluation  items  . Of  the  ten  comments  provided  by 
graduate  students,  three  students  felt  that  the  pace  of  the  program  was  too  slow  to  meet  their  needs  and  two 
students  indicated  that  only  some  of  the  content  was  new  to  them.  The  undergraduate  students  provided  many 
more  comments  (n  = 86)  and  were  more  positive  in  their  evaluation.  The  two  most  frequent  comments  had  to  do 
with  ease  of  use  (n  = 6)  and  enhancement  of  current  knowledge  («  = 6).  Five  comments  indicated  that  not  all  of 
the  content  was  new  to  the  student.  However,  it  appeared  that  SPARK  was  able  to  either  reinforce  information 
that  students  were  unsure  about  or  that  it  corrected  misinformation. 

The  amount  of  time  spent  in  SPARK  by  undergraduate  students  as  compared  to  graduate  students  was 
significantly  higher.  Several  circumstances  may  account  for  the  difference.  The  undergraduate  students  were 
taking  a class  from  one  of  the  investigators  (Hrabe);  they  also  received  extra  credit  for  taking  the  time  to 


Undergraduate 

Graduate 

Overall 

Mean 

Mean 

Mean 

4.53 

4.89 

4.70 

3.11 

2.50 

2.81 

4.11 

3.77 

3.95 

3.84 

3.94 

3.89 

3.63 

3.94 

3.78 

3.58 

3.61 

3.59 

45.79 

29.64 

38.94 
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complete  an  online  survey.  The  graduate  students  completed  a paper  and  pencil  survey  voluntarily  (i.e.,  no  extra 
credit)  and  the  investigators  were  unknown  to  this  group.  The  positive  evaluations  could  also  reflect 
participants’  gratitude  for  receiving  a free  copy  of  a CD  and  faculty  concern  for  the  students’  success  in  school. 


Discussion  and  Summary 

Overall,  data  suggest  a positive  experience  with  SPARK.  Ratings  indicate  that  students’  felt  the  CD 
was  easy  to  use,  kept  their  attention  and  enhanced  their  confidence  in  learning  the  skills  necessary  to  navigate 
online  courses.  While  the  lowest  rankings  indicated  that  much  of  the  content  was  not  new  to  the  participants, 
having  the  information  readily  available  helped  to  refresh  and  reinforce  what  they  already  knew  and  increased 
their  confidence. 

Using  SPARK  or  similar  approaches  highlights  the  importance  of  helping  students  acquire  the 
technical  expertise  they  need  to  be  successful  in  hybrid  or  totally  online  courses . These  endeavors  should  assist 
faculty  in  narrowing  the  gap  between  the  skills  students  bring  versus  those  they  need . Future  work  will  focus  on 
improving  assessment  of  skill  and  matching  results  to  targeted  remediation. 
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Table  3.  Selected  Screens  from  SPARK,  copyright  and  patent  pending  2004. 


Figure  1. 

SPARK  Title  Screen 


Figure  2. 

Introductory  screen  asks  user  to  rate  his  or 
her  computer  skills . Narrated  voice-over 
feedback  is  individualized  according  to 
response. 


Figure  3. 

Program  Navigation  Instructions  includes 
voice-over  narration. 
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Figure  4. 

SPARK  Topic  Map  allows  random 
navigation  to  any  topic  or  sub-topic. 


Figure  5. 

This  instruction  screen  from  the  Hardware 
category  is  about  DVD  storage  capacity 
versus  CD  capacity.  Additional  sub-topics 
are  offered  on  the  left. 
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It  Is  More  about  Telling  Interesting  Stories:  Use  Explicit  Hints  in 
Storytelling  to  Help  College  Students  Solve  Ill-defined  Problems 


Wen-Lan  Hsieh 
Brian  K.  Smith 
Spiro  E.  Stefanou 
Penn  State  University 

Abstract 

A team  consisting  of  three  faculty  members  from  Agricultural  Economics,  Agribusiness  management, 
and  Food  Science  with  two  research  assistants  at  Penn  State  University  has  been  working  for  three  years  on 
creating  a food  product  case  library  for  a problem-based  learning  and  case-based  instruction  course.  With  the 
assistance  of  experts  from  the  food  manufacturing  and  retailing  industries  we  collected  approximately  1 10 
stories  related  to  food  product  development.  These  stories  were  organized  and  stored  into  a database  (a  Case 
Library ) for  faculty  and  students  to  use  in  a case-based  instruction  course. 

An  earlier  research  study  conducted  by  our  team  members  found  evidence  that  a Case  Library  with 
stories  did  affect  students  ’ decisions  of  making  multiple-choice  tests  concerning  ill  structured  problems 
(Hernandez-Serrano,  2001 ).  Then,  we  encountered  that  students  had  difficulty  making  on-point  connections 
between  stories  and  target  problems  at  hand.  Our  goal  is  to  help  learners  better  understand  the  stories  while 
enhancing  their  abilities  to  make  analogies.  This  has  raised  the  following  questions:  What  should  we  do  to 
achieve  this  goal ? What  kind  of  story-indexing  strategies  will  help  students  understand  the  stories  better?  Will 
surface  level  indices  (such  as  company  name,  product  category,  product  name,  and  development  process)  help 
students  recall  similar  features  easier?  Will  a deeper  level  of  indices  (such  as  theme,  goal,  plan,  results,  and 
lesson)  help  students  understand  the  stories  better? 

Problem  Statement  and  Research  Question 

Given  the  importance  of  ill -structured/ill -defined  problem  solving  in  the  workplace,  instructional 
materials  and  activities  should  be  situated  in  a contextual  learning  environment.  An  ill-structured  problem  may 
lack  a clear  initial  state,  a set  of  permissible  operators,  or  a clear  goal  state  (Chi  & Glaser,  1985).  In  addition, 
there  are  no  absolute  correct  answers,  so  this  type  of  problem  lends  itself  to  multiple  viable  solutions  making  it 
difficult  to  teach  students  how  to  solve  ill-structured  problems.  Research  has  shown  that  stories  are  more 
memorable,  promote  elaboration  to  personal  experiences,  and  help  in  solving  ill-structured  proble  ms  (Swap, 
Leonard,  Shields,  and  Abrams,  2001;  Hernandez-Serrano,  et  al.  2002).  However,  research  has  not  shown 
whether  academic  achievement  of  solving  ill-structured  problems  is  improved  by  using  stories  with  pre- 
generated  story  indices  developed  by  experts  in  the  representative  field. 

Based  on  the  research  purpose  stated  above,  answers  to  the  following  questions  are  sought: 

Qi:  What  are  the  effects  of  using  different  story -indexing  strategies  within  a Case  Library  on  college  students’ 
(novice  learners’)  ability  to  solve  ill-structured  problems? 

Q2:  How  do  novice  learners  make  analogies  between  stories  and  targeted  problems  when  they  solve  ill- 

structured  problems?  What  processes  are  used  when  novice  learners  read,  encode,  retrieve,  and  adapt  the 
source  stories  to  the  targeted  problems? 

Q3:  How  do  pre-selected  indices  help  novice  learners  see  similar  features  between  the  stories  and  the  targeted 
problems? 


Literature  Review 

Problem  Solving:  Experts  classify/  index  problems  differently  from  novices 

Chi  & Glaser  (1985)  describe  two  factors  influencing  people’s  problem  solving  abilities:  the  kind  of 
knowledge  brought  to  the  problem  by  the  solver  and  the  nature  of  the  task.  The  knowledge  brought  to  the 
problem  by  the  solvers  varies  depending  on  the  amount  of  their  knowledge  of  specific  domain  content. 

Identifying  differences  between  experts  and  novices  offers  a key  to  understanding  problem  solving 
processes.  Some  studies  show  that  novices  tend  to  classify  problems  by  their  surface  structure,  focusing  on  the 
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words  or  subjects  that  are  prominent  in  the  problem  statements;  some  studies  indicate  that  experts  appear  to 
base  their  perceptions  of  problem  relatedness  upon  problems’  deep  structure. 


Case-based  Reasoning  and  Story  Index 

The  application  of  storytelling  to  problem-solving  skill  is  supported  by  case-based  reasoning  (CBR),  a 
learning  theory  focused  on  analogy  in  the  context  of  solving  real-world  problems  by  encoding,  retrieval,  and 
adaptation  in  analogical  reasoning  process  (Kolodner,  1997,  p.57).”  Suggested  by  CBR,  a Case  Library  is  built 
to  provide  the  resources  of  cases  by  collecting  stories  from  experts.  A Case  Library  is  a systematic  collection 
and  organization  of  a number  of  experts’  experiences  presented  in  the  form  of  stories  to  the  learner  as  they 
interact  with  a task  environment  (Edelson,  1993).  See  Figure  1 for  an  example  of  the  Food  Product  Case 
Library. 

Indexing  is  the  process  of  assigning  labels  to  stories  based  on  specific  rules  or  interpretations  when 
putting  stories  into  a Case  Library  (Kolodner  1993).  Indexing  consists  of  “labeling”  an  experience  with  the 
appropriate  “title”  and  then  “filing”  it  in  the  right  place  in  memory,  which  is  the  process  of  organizing 
experiences  so  that  people  know  where  they  can  find  relevant  information  when  needed  (Schank,  Berman,  and 
Macpherson,  1999). 


Figure  1 : Food  Product  Case  Library 
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A New  Opportunity  - Defining  the  idea  and  Concept 

I n the  early  80's,  Giant  Food,  a regional  supermarket  chain, 
commissioned  surveys  and  focus  groups  that  revealed  a demand  for 
home  meal  replacement  items  in  their  produce  departments.  As  a 
result,  they  introduced  a vegetable  platter  prepared  in-store  that  was 
marketed  as  a holiday  item. 


I n the  early  90's,  they  began  to  look 
for  similar  alternatives  in  the  fruit 
category.  Concomitantly, 
supermarkets  were  seeking  ways  to 
market  fruit  that  was  damaged  or 
overripe.  As  a way  to  salvage  the 
bruised  fruit  and  satisfy  consumer 
demand,  the  stores  began  peeling 
and  cutting  the  fruit,  putting  it  in  deli 
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Explicit  Story  Indices  and  Problem  Solving 

Novices  lack  experience  to  draw  on;  even  if  they  do  have  experience,  they  may  have  difficulties  using 
these  experiences  well  because  they  lack  a good  understanding  of  how  to  encode  their  experiences,  are  unable 
to  make  retrieval  at  appropriate  times,  and  cannot  reuse  experiences  (Kolodner,  1997).  In  contrast  to  novices, 
experts  have  many  experiences  in  their  areas  of  expertise  stored  in  their  “library”  of  memory,  and  they  can 
retrieve  the  right  story  to  solve  new  problems  (Schank,  Berman,  and  Macpherson,  1999).  An  expert  is  someone 
who  has  a great  many  stories  to  tell  in  one  particular  area  of  knowledge  and  who  has  those  stories  indexed  well 
enough  to  find  the  right  one  at  the  right  time  (Schank,  1995).  Schank  (1995)  tried  three  different  methods  of 
extracting  stories  from  an  expert  who  proved  to  be  a repository  of  stories  about  various  episodes  in  military 
history.  He  can  see  military  stories  in  a variety  of  different  ways  because  he  has  created  for  himself  a set  of 
complex  indices  about  military  history. 

In  the  experimental  study  of  Gick  and  Holyoak  (1980,  1983)  and  Holyoak  (1990),  college  students 
were  asked  to  use  the  fortress  problem  to  help  solve  an  ill-defined  radiation  problem.  The  desired  goal  of  the 
radiation  problem  is  specified  at  an  abstract  level  and  the  strategies  used  to  achieve  the  goal  are  open  ended. 
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Without  the  fortress  problem  as  source  analog,  very  few  students  proposed  the  idea  of  using  low  intensity  rays. 
For  those  who  received  fortress  problems  as  a resource  analogy,  they  performed  differently  in  generating 
solutions.  When  a hint  to  use  the  story  is  provided,  most  of  the  students  came  up  with  the  idea  of  using 
convergent  low  intensity  rays;  when  students  are  not  told  to  apply  the  prior  fortress  story  to  help  solve  the 
radiation  problem,  their  transfer  performance  of  generating  analogous  solutions  declined  Why  did  most  of  the 
subjects  fail  to  notice  the  relevance  of  a story  analogy  to  a target  problem?  It  wasconcluded  that  the  difficulty 
may  be  related  to  the  problem  of  identifying  the  optimal  level  of  abstraction  for  representing  the  similar 
features. 

Novices  without  enough  domain  knowledge  and  cases  stored  in  memory  only  see  the  surface  features 
when  asked  to  solve  problems  and  interpret  problems  but  experts  classify  problems  by  principles  and  specific 
rules.  How  do  we  gather  and  organize  expert  stories  to  help  novices  learn?  The  application  of  storytelling  to 
problem-solving  skills  is  supported  by  case-based  reasoning  (CBR)  and  a Case  Library  is  built  to  provide  the 
resources  of  cases  by  collecting  stories  from  experts. 

If  using  cases  as  a resource  analogy  is  such  a natural  and  efficient  way  to  help  novice  learners 
understand  problem  situations  and  propose  solutions,  how  do  we  help  novice  learners  see  the  relevance?  How 
do  we  help  them  see  more  than  surface  features? 

Our  hypothesis  is  that  using  stories  collected  from  experts  and  indexing  them  by  deeper  features/ 
thematic  features  can  make  these  connections  explicit  for  novice  learners. 

Method 

This  research  is  a mixed-method  design  with  a quantitative  design  (control,  comparable,  and 
experimental  group)  and  a qualitative  design  (think-aloud  protocol).  The  different  treatments  for  the  three 
groups  are  listed  below. 

1.  Control  group:  stories  are  not  indexed  with  any  labels  or  hints. 

2.  Comparable  group:  stories  are  indexed  with  surface  indices  (factual  information,  such  as  the  name 
of  the  product,  the  name  of  the  company,  the  category  of  the  product,  and  the  process  of  the 
product  development). 

3.  Experimental  group:  stories  are  indexed  with  belief-based  indices  according  to  Schank’s  (1990). 
Belief-  based  indices  include  theme,  goal,  plan,  result,  and  lesson. 

See  Figure  2 for  an  example  of  story  with  surface  indices  and  Figure  3 for  an  example  of  story  with 
thematic  indices. 


Figure  2 : Story  with  surface  indices 
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Just  "bottled  water" 

Just  a few  years  ago,  when  most  people  wanted  to  have  a drink  of  water, 
most  likely  they  turned  to  a water  fountain  in  the  school,  work  or  home.  Nowadays  it 
is  more  common  to  see  the  upscale  crowd  carrying  bottled  water.  This  phenomenon 
seems  to  be  driven  by  demographics.  People  are  becoming  more  health  conscious. 
The  attitude  seems  to  be  "I  don't  drink  tap  water  because  is  chlorinated  and  I don't 
trust  it.  It's  not  good  for  my  health." 

Trying  to  cash  in  on  this  craze  around  bottled  water,  the  PepsiCo  corporation  launched  its  popular  water 
product  AquaFina.  The  marketing  managers  behind  this  product  knew  that  this  demographic  group  would  be 
willing  to  pay  a certain  price  for  this  product  if  it  matched  consumers'  notions  and  expectations  of  health.  That 
has  been  paired  to  powerful  images  of  spring  waters  from  Colorado  and  France  further  highlighting  notions  about 
health,  freshness  and  purity.  The  product  has  been  a success. 

The  AquaFina_product  has  been  welkpositioned  against  a demographic  proup.  The  PepsiCo  corporation 
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has  been  successful  by  correctly  applying  demographic  data  to  position  a product  in  the  consumer's  mind,  thus 
meeting  the  expectations  of  the  more  health  conscious  consumer  of  toda_y. 


Figure  3:  Story  with  thematic  indices 
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Result 

Successfully  applying  demographic  data  to  position  a product 

Lesson 

Evaluating  demographic  trends  can  develop  new  product  concepts 

Just  "bottled  water" 

Just  a few  years  ago,  when  most  people  wanted  to  have  a drink  of  water, 
most  likely  they  turned  to  a water  fountain  in  the  school,  work  or  home.  Nowadays  it 
is  more  common  to  see  the  upscale  crowd  carrying  bottled  water.  This  phenomenon 
seems  to  be  driven  by  demographics.  People  are  becoming  more  health  conscious. 
The  attitude  seems  to  be  "I  don't  drink  tap  water  because  is  chlorinated  and  I don't 
trust  it.  It's  not  good  for  my  health." 

Trying  to  cash  in  on  this  craze  around  bottled  water,  the  PepsiCo  corporation  launched  its  popular  water 
product  AquaFina.  The  marketing  managers  behind  this  product  knew  that  this  demographic  group  would  be 
willing  to  pay  a certain  price  for  this  product  if  it  matched  consumers'  notions  and  expectations  of  health.  That 
has  been  paired  to  powerful  images  of  spring  waters  from  Colorado  and  France  further  highlighting  notions  about 
health,  freshness  and  purity.  The  product  has  been  a success. 

The  AquaFina  product  has  been  well  positioned  against  a demographic  group.  The  PepsiCo  corporation 
has  been  successful  by  correctly  applying  demographic  data  to  position  a product  in  the  consumer's  mind,  thus 
meeting  the  expectations  of  the  more  health  conscious  consumer  of  todaj. 


The  main  purpose  of  this  quantitative  design  is  to  demonstrate  the  effect  of  using  different  story  - 
indexing  strategies  within  a Case  Library  on  novice  learners’  abilities  of  solving  ill -structured  problems.  By 
manipulating  the  variable  of  story -indexing  strategy  (grouping  participants  into  control,  comparable,  and 
experimental  groups  and  by  allowing  each  group  access  to  a Case  Library  with  either  no  indices,  surface 
indices,  or  thematic  indices),  we  can  gauge  the  effect  on  learners’  performance  scores  on  solving  ill-structured 
problems.  The  main  purpose  of  the  qualitative  design  is  to  gather  verbal  reports  from  the  novice  learners  when 
they  are  taking  the  open-ended  test  in  order  to  investigate  and  understand  the  process  of  problem  solving  and 
analogy  making. 

All  subjects  were  selected  from  university  junior  or  senior  students  taking  related  courses  in  marketing 
management,  food  marketing  management,  or  agriculture  business.  They  took  one  training  session  and  one 
open-ended  test  session  in  this  study.  Those  participating  in  the  think-aloud  activity  attended  both  sessions.  The 
main  difference  is  they  have  to  talk  aloud  their  thinking  process  while  solve  problems.  The  entire  think-aloud 
process  is  video-  and  audio -taped  and  used  as  qualitative  data.  See  Figure  4 for  the  flow  chart  of  quantitative 
design;  see  Figure  5 for  the  flow  chart  of  qualitative  design. 

For  the  open-ended  test,  a rubric  with  scoring  rules  was  created  by  the  first  author  and  a professor  of 
Agricultural  Economics.  Two  raters  used  the  rubric  to  assign  scores  for  students’  answers  on  those  open-ended 
questions.  The  scores  were  treated  as  quantitative  data.  Therefore,  each  student  has  two  scores  from  two  raters. 
Average  of  the  scores  was  used  to  see  if  there  is  any  significant  difference  due  to  the  treatment.  For  the 
qualitative  data,  the  data  were  coded,  categorized,  and  analyzed  by  following  the  method  of  verbal  data  analysis 
from  Chi  (1997). 
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Anticipated  Outcomes 

It  is  expected  that  the  group  with  access  to  a number  of  stories  with  thematic -indexing  strategy  in  a 
Case  Library,  will  perform  better  on  an  open-ended  test  evaluating  high-order  thinking  skills  of  solving  ill- 
structured  problems  than  a comparable  group  who  have  access  to  stories  with  surface-indexing  strategy  and  a 
control  group  who  has  access  to  stories  without  indexing  strategy.  It  is  also  expected  that  through  the  careful 
analysis  of  learners’  verbal  reports,  the  nature  and  process  of  analogical  problem  solving,  the  transferring 
process  from  experts’  experiences/stories  to  novice  learners,  and  the  construction  and  function  of  personal 
indexing  schema  will  be  revealed. 


Figure  4 : A flow  chart  of  the  quantitative  design 
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Figure  5:  A flow  chart  of  the  think-aloud  protocol 
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Impact  of  Cognitive -based  Instructional  Intervention  on  Learning 
Motivation:  The  Implementation  of  Student-made  Glossary  in  a 
Programming -oriented  Engineering  Problem-solving  Course  and  Its 

Impact  on  Learning  Motivation 

David  Wenhao  Huang 
Hans  Aagard 
Heidi  Diefes-Dux 
Purdue  University 

Abstract 

This  article  describes  the  purpose,  development,  and  implementation  of  a cognitive-based  instructional 
intervention  and  its  impact  on  learning  motivation.  The  study  was  conducted  in  a programming-based  problem- 
solving course  for  first -year  engineering  students.  The  results  suggest  that  the  instructional  intervention 
developed  based  on  the  hierarchical  analysis  of  intellectual  skills  development  and  partial-to-whole  learning 
task  approach  has  significant  correlation  with  the  Satisfaction  component  of  ARCS  Motivational  Design  Model. 

Introduction 

Problems  with  current  approaches  to  study  of  motivation 

Motivation  is  a critical  factor  in  learning  (Linnenbrink  & Pintrich,  2002)  and  many  researchers  have 
examined  the  subject  from  various  perspectives  (Gonzalez- Pienda,  et  al.,  2002;  Hancock,  2002;  Cokley,  et  al., 
2001).  Three  issues  trouble  current  studies  of  learning  motivation:  the  lack  of  a systematic  approach,  neglect  of 
other  aspects  of  learning,  and  the  inability  to  separate  different  aspects  of  motivation. 

The  affective  and  complex  nature  of  motivation  makes  it  is  difficult  to  study,  resulting  in  many 
motivational  studies  that  lack  a systematic  approach.  This  lack  of  sytematicity  limits  reproduction  of  the  effect 
and  makes  application  difficult  in  practical  situations.  A review  of  motivational  studies  based  on  various  aspects 
of  learning  (e.g.  cognitive,  social,  and  attitudinal)  reveals  that  there  is  no  sound  systematic  approach  applied  to 
them;  most  are  solely  based  on  motivational  theories  (Harlen  & Crick,  2004)  and  frameworks  to  establish  a 
broad,  inductive  goal  without  identifying  a motivational  problem.  Identifying  the  motivational  problem  can 
clarify  the  motivational  analysis  and  intervention.  For  example,  Bandura’s  Self-Regulation  theory  is  well 
received  in  the  learning  motivation  field  and  efforts  are  being  made  to  identify  the  measurable  variables 
affecting  levels  of  Self -Regulation  (Miller  & Brickman,  2004).  However,  there  is  no  suggestion  as  to  how  those 
variables  or  components  can  be  practically  applied  to  resolve  specific  problems. 

In  other  words,  one  problem  with  learning  motivation  studies  is  a lack  of  linkage  between  theories  and 
practical  instructional  practices.  Many  theorists  have  proposed  theories  on  how  people  get  motivated  and  what 
behaviors  can  be  stimulated  by  motivation  (Weiner,  1985;  Ames,  1992;  Anderman  & Maehr,  1994;  Bandura, 
1997).  However,  educators  are  still  in  search  of  practical  and  applicable  guidelines  that  would  enable  them  to 
convert  motivation-enhancing  theories  into  practical  instructional  practices  with  confidence  (Hancock,  2002). 

Another  problem  with  current  studies  of  learning  motivation  is  their  narrow  approach.  Given  the 
complexity  of  investigating  and  measuring  learning  motivation  in  any  instructional  setting,  various  aspects  of 
learning  should  be  considered  when  trying  to  address  motivational  issues  holistically.  Aspects  ranging  from 
internal  factors  like  student  learning  styles  to  external  factors  like  the  learning  environment  and  applied 
instructional  strategies  should  be  considered. 

Finally,  the  other  learning  components,  such  as  instructional  strategies,  along  with  affective 
components  traditionally  associated  with  motivation  such  as  enthusiasm  and  understanding  of  content,  need  to 
be  separated  and  their  effects  on  each  other  evaluated.  This  poses  the  concern  of  how  reliable  and  valid 
motivational  studies  are  if  multiple  issues  are  addressed  simultaneously.  Because  factors  affecting  learning  do 
interact  with  each  other,  a confounding  effect  can  occur.  Motivation  does  not  occur  in  a vacuum  and  the  effects 
other  factors  in  learning  have  on  motivation  are  traditionally  neglected  (Astleitner  & Wiesner,  2004). 

A Systematic  Approach:  Selection  of  the  ARCS  Model 

In  order  for  the  results  of  motivational  studies  to  be  more  practical  in  addressing  known  instructional 
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or  learning  problems,  perhaps  a problem-solving  process  should  accompany  the  investigation  process.  A 
generic  instructional  design  process  whose  components  are  found  in  many  instructional  design  models  (e.g. 

Dick,  1996;  Smith  & Ragan,  1993;  Gentry,  1994,  Gustafson  & Branch,  1997)  may  be  an  appropriate  tool.  The 
ADDIE  process  is  geared  towards  solving  instructional  problems  in  five  general  steps;  (A)  analysis  of  the 
instructional  problem,  (D)  design  and  (D)  development  of  the  instructional  intervention,  (I)  implementation  of 
the  intervention,  and  (E)  evaluation  of  the  outcome(s).  These  systematic  steps  can  be  applied  to  the 
investigation  of  learning  motivation  due  to  their  generalizability. 

A motivational  model  with  practical  instruments  should  be  used  along  with  a problem-solving  process 
to  address  specific  aspects  of  motivation  that  a broad  instructional  design  model  like  ADDIE  cannot.  The  ARCS 
model,  a motivational  design  model  (Keller,  1987a,  b),  provides  both  a theoretical  framework  and  a tool  for 
assessing  motivational  levels  by  following  the  steps  of  ADDIE  The  ARCS  model  suggests  that  learning 
motivation  is  influenced  by  four  components:  Attention,  Relevance,  Confidence,  and  Satisfaction.  By  enhancing 
an  individual  component  using  specific  motivational  strategies  and  instructional  methods,  students’  learning 
motivation  can  be  improved.  Keller  (1987b)  suggested  a systematic  approach  to  first  identify  motivational 
problems  and  then  prescribe  motivational  strategies  to  solve  them.  A pre  - motivational  survey  is  needed  to 
establish  students’  initial  motivational  level.  Keller’s  (1993)  Instructional  Materials  Motivational  Survey 
(IMMS)  was  developed  as  a situated  measuring  instrument  to  gauge  the  learning  motivation  of  specific 
instructional  materials.  From  the  survey,  motivational  strategies  can  be  prescribed  according  to  identified 
motivational  objectives.  A post-motivational  survey  should  be  conducted  to  examine  the  effectiveness  of  the 
motivational  strategies  applied. 

Motivational  Studies  Focusing  on  Cognitive  Information  Processing 

Another  way  to  make  motivational  studies  more  widely  applicable  is  to  focus  on  the  cognitive  side  of 
learning.  Studies  on  the  cognitive  aspects  of  learning,  including  cognition  development,  have  established  a 
scientific  methodology  to  empirically  investigate  the  human  learning  process  from  an  information  processing 
viewpoint.  This  deductive  approach  makes  the  studies  more  replicable,  the  focus  of  research  questions  more 
clear,  and  the  outcomes  more  reliable  and  easier  to  identify. 

In  addition  to  the  measurable  outcomes  these  scientific  research  designs  can  produce,  various  studies 
have  suggested  a possible  relationship  between  cognitive  information  processing  and  learning  motivation 
(Wolters,  2004;  Chalupa,  Chen,  & Charles,  2001).  For  example,  Malone  and  Lepper  (1987)  proposed  four 
components  to  establish  learning  motivation,  all  derived  from  human  mental  cognition.  The  association  between 
motivation  and  cognition  is  also  supported  by  the  Expectancy  Theory  (Vroom,  1964)  and  the  Control  Theory 
(Klein,  1989)  in  which  the  learner’s  goal  setting  behaviors  and  perceived  control  are  emphasized.  Astleitner  and 
Wiesner  (2004)  further  proposed  an  integrated  model  of  multimedia  learning  and  motivation  that  describes  the 
relationship  between  memory  capacity  and  resource  management,  and  their  effect  on  learning  motivation. 

From  an  instructional  design  viewpoint,  the  systematic  process  to  approach  motivational  problems  and 
the  relationship  between  cognitive  processing  and  motivation  means  it  is  possible  to  design  and  develop 
interventions  for  motivational  problems  based  on  a cognitively  based  instructional  strategy. 

Purpose  of  the  Study 

The  purposes  of  this  study  are  to  examine  the  feasibility  of  adopting  a systematic  and  instructional 
design-oriented  research  design  for  the  investigation  of  learning  motivation,  and  to  investigate  the  relationship 
between  a cognitively  based  intervention  and  learners’  motivation.  This  study  employed  (1)  a procedure  to 
diagnose,  analyze,  and  develop  instructional  interventions  for  better  motivational  outcomes,  (2)  a hierarchical 
analysis  of  intellectual  skills  to  better  understand  the  complexity  of  the  subject’s  cognitive  learning  tasks,  and 
(3)  the  cognitive  load  theory  (Sweller,  1994)  as  the  foundation  for  developing  the  intervention  in  response  to  the 
identified  motivational  problems. 


Methodology 


Setting  and  Participants 

Our  focus  in  this  study  was  on  how  subjects  used  a computer-based  tutorial  called  M-Tutor™.  M- 
Tutor  was  designed  to  help  students  learn  MATLAB®  syntax.  MATLAB®  is  a computational  software  package 
that  integrates  mathematical  computing,  visualization,  and  a powerful  comp  uter  programming  language  to 
provide  a flexible  environment  for  technical  computing  (Mathworks,  2003). 
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In  the  Fall  semester  of  2002,  first-year  engineering  students  who  enrolled  in  ENGR  106,  Engineering 
Problem-Solving  and  Computer  Tools,  were  instructed  to  use  M-Tutor  as  their  primary  means  of  learning 
MATLAB  syntax.  This  2 credit-hour,  required  course  is  designed  to  develop  first-year  engineering  students’ 
abilities  to  solve  engineering  problems  with  appropriate  computer  tools.  In  order  to  accomplish  the  course 
objectives,  the  instructor  needed  to  create  an  active  learning  environment  in  the  lecture,  which  emphasizes 
fundamental  engineering  concepts  and  problem-solving  strategies.  However,  the  students  needed  to 
simultaneously  learn  to  be  efficient  and  effective  users  of  computer  tools  and  use  those  tools  to  solve 
engineering  problems.  The  instructor  struggled  with  devoting  class  time  to  simply  telling  students  about 
computer  tools  and  how  they  worked  versus  covering  fundamental  engineering  concepts  and  how  to  use  these 
computer  tools  to  solve  realistic  problems.  Thus  the  instructor  adopted  M-Tutor  to  help  students  learn 
MATLAB  syntax  outside  of  class.  Students’  motivation  for  using  M-Tutor  to  learn  MATLAB  was  the  focus  of 
this  study. 

Keller’s  ARCS  Model  of  Motivational  Design  and  Instructional  Materials  Motivational  Survey 
(IMMS)  were  adopted  as  the  theoretical  framework  of  learning  motivation  as  well  as  the  basis  for  quantitative 
and  qualitative  data  collection.  The  study  employed  the  one  group  pre  and  post-tests  design.  Three  surveys  were 
implemented:  Pre -motivational  survey,  post-motivational  survey,  and  an  additional  survey  on  the  use  of 
student -made  glossary.  The  treatment  was  having  students  complete  instructor-guided  glossary  items.  The  last 
survey  focused  on  the  glossary  itself  was  implemented  in  order  to  associate  students’  perception  towards  the  use 
of  the  glossary  with  their  learning  motivation  levels.  The  pre -motivational  survey  based  on  the  ARCS  Model 
indicated  that  students  gave  a relatively  low  rating  on  the  Satisfaction  component.  Students’  qualitative 
responses  also  suggested  that  they  felt  frustrated  when  using  the  computer-based  tutorial.  The  main  reason  for 
that  reaction  is  the  lack  of  connection  between  isolated  coding  tasks  and  their  application  to  engineering 
problems.  Thus  the  motivational  strategy  was  developed  based  on  the  pre -motivational  survey  with  specific 
emphasis  on  the  Satisfaction  component  of  ARCS  Model. 

The  strategy  for  enhancing  students’  Satisfaction  level  is  to  provide  students  with  more  opportunities 
to  gain  a sense  of  accomplishment  on  course  assignments  by  using  M-Tutor™.  Providing  immediate  feedback 
is  also  considered  a crucial  element  for  better  motivational  outcome.  Thus  the  student -made  glossary 
assignment  was  developed  as  the  intervention  to  carry  out  the  motivational  strategy,  which  allows  students  to 
receive  meaningful  feedback  as  well  as  obtain  a sense  of  accomplishment  prior  to  solving  application  problems. 
The  feeling  of  accomplishment  is  obtained  by  allowing  students  to  go  through  smaller  parts  of  the  learning  task 
(i.e.  individual  syntax  as  opposed  to  a chunk  of  coding).  There  are  five  columns  on  the  glossary  form:  (1) 
syntax,  (2)  overview  of  the  syntax,  (3)  student  developed  test  case(s)  based  on  instructor  guidelines,  (4)  hand 
computations  to  predict  results  of  test  cases,  and  (5)  MATLAB  results.  The  complexity  level  of  each  column  is 
increased  by  following  the  hierarchical  analysis  of  intellectual  skills  (i.e.  discrimination,  concepts,  and  rules). 

By  accomplishing  the  lower  levels  allows  students  to  build  a schema  and  tie  the  ideas  together,  so  that  when 
they  are  presented  with  a problem,  they  can  relate  the  problem  to  their  new  schema. 

Each  glossary  item  was  designed  by  the  instructor  according  to  immediate  assignments  since  it  helps 
students  to  better  transfer  newly  acquired  programming  skills  (composing  with  syntax)  to  application  problems. 
Students  were  asked  to  complete  the  glossary  before  they  worked  on  application  problems  . 

Research  Design 

This  study  included  all  five  steps  of  the  generic  instructional  design  process  discussed  earlier  (i.e. 

ADDIE). 

(1)  The  analysis  of  the  pre -motivational  survey  and  examining  its  data. 

(2)  The  design  of  motivational  intervention  based  on  the  pre -motivational  data. 

(3)  The  development  of  a motivational  intervention  based  on  previous  relevant  research  and  theories  as 
well  as  subject  matter  experts’  input. 

(4)  The  implementation  of  the  developed  motivational  intervention  (instructional  strategy)  to  the  actual 
instructional  setting. 

(5)  The  evaluation  of  the  motivational  strategy  by  a post-motivational  survey  and  an  intervention-specific 
survey. 

Development  of  Motivational  Strategies  and  Data  Processing 

Pre-  and  Post-Motivational  Survey  and  Data  Analysis  The  first  motivational  survey  was  conducted  in 
the  week  after  the  first  M-Tutor  assignment  while  the  post-motivational  survey  was  given  after  the 
implementation  of  intervention.  The  survey  instrument  collected  students’  Pre-  and  Post-  reactions  towards  the 
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tutorial.  Keller’s  IMMS  survey  was  adapted  in  order  to  accommodate  the  computer-based  study  setting.  It 
consisted  of  36  rating  questions  (Cronbach’s  Alpha=  0.917).  The  Instructional  Materials  Motivational  Survey 
(IMMS)  (Keller,  1993)  was  developed  based  on  the  ARCS  Model.  The  IMMS  consists  of  36  statements  that  are 
rated  using  Likert -type  scales  (1  = Not  True;  5 = Very  True).  Each  item  is  mapped  to  an  individual  ARCS 
component  and  provides  a measure  of  the  respondent’s  perception  of  that  particular  component.  Quantitative 
data  are  composed  of  ratings  from  the  IMMS.  It  is  important  to  re  member  that  responses  to  a Likert -type  scale 
item  generate  categorical  data  that  cannot  be  averaged  to  provide  a mean  response  for  an  individual  survey  item 
Therefore,  the  frequency  with  which  students  responded  "Mostly  True"  and  "Very  True"  on  individual  survey 
items  were  computed.  For  all  items  mapped  to  a particular  ARCS  component,  the  frequency  with  which 
students  responded  "Mostly  True"  (4)  and  "Very  True"  (5)  was  used  to  provide  a single  quantitative  measure  of 
that  ARCS  component.  Qualitative  data  was  also  collected  by  open-ended  questions  attached  to  the  survey.  The 
main  purpose  of  collecting  qualitative  data  was  to  better  identify  design  issues  within  each  instructional 
component. 

Studies  indicate  that  the  ARCS  Model  is  applicable  in  the  computer-based  or  web-based  instructional 
environment  (Keller,  1999;  Keller  & Song, 1999;  Knowlton  , Shellnut  & Savage,  1999;  Park&  Hannifin,  1993) 
although  it  was  originally  designed  for  developing  motivating  instructional  materials  in  traditional  face-to-face, 
classroom  settings.  For  this  pilot  study,  the  IMMS  was  modified  to  assess  the  motivational  effectiveness  of  M- 
Tutor.  Each  survey  item  was  revisited  and,  as  needed,  re -focused  on  the  research  question,  which  was  to 
diagnose  students’  motivational  level  in  using  the  tutorial  as  a learning  tool,  with  the  expectation  that  students 
would  effectively  learn  MATLAB  syntax  and  effectively  use  MATLAB  as  a tool  for  solving  engineering 
problems. 

Design,  Development,  Implementation,  and  Evaluation  of  Motivational  Strategies  After  the  pre- 
motivational  survey,  we  analyzed  the  quantitative  as  well  qualitative  data  to  determine  which  aspect  of 
motivation  to  address.  A coding  system  (Table  1)  was  developed  for  analyzing  qualitative  data  based  on  design 
principles  of  multimedia  courseware  (Szabo  & Kanuka,1998;  Evans  & Edwards,  1999;  Coscarelli  & Shrock, 
2000).  The  instructional  coding  system  categorizes  qualitative  responses  from  each  mapped  ARCS  Model  open- 
ended  question  into  various  instructional  components . The  instructional  components  involved  were  coded  as 
interface  design,  content,  learning  support,  and  implementation.  Each  qualitative  response  could  be  coded  in 
one  or  more  categories.  The  research  direction  was  visited  repeatedly  during  the  development  of  the  coding 
system  to  insure  the  validity  of  items  (Coscarelli  & Shrock,  2000). 

By  triangulating  the  quantitative  and  qualitative  data,  valuable  information  can  be  gained.  First,  the 
research  team  can  identify  which  instructional  component(s)  are  most  influential  on  students’  motivational 
levels.  Second,  the  research  team  can  map  the  instructional  components  of  the  tutorial  (interface,  content, 
learning  support,  and  implementation)  to  the  ARCS  Model  components  (Attention,  Relevance,  Confidence,  and 
Satisfaction). 

Also  to  explicitly  evaluate  the  instructional  intervention,  a ten-question  survey  was  developed  to 
measure  students’  reactions  towards  the  intervention  independently  of  its  effectiveness  in  enhancing  their 
motivational  level.  (Cronbach’s  Alpha  =0.926,  N=957). 

Table  1.  EXAMPLES  OF  CODES  AND  INTERPRETATIONS 


Instructional 
Component  Code 

Keyword  Examples 

Interface  Design 

Text,  graphs,  navigational  system,  interactivity 
menu 

Content 

Relevance,  easiness  and  difficulty  of  information 

Learning  Support 

Textbook,  assistance,  Help  session,  feedback, 
exercises 

Implementation 

Course  structure,  technical  infrastructure 
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Results 


Pre-motivational  survey  analysis 

Table  2 indicates  that  students  initially  provided  the  least  of  amount  of  positive  response  towards  the 
Satisfaction  component  of  ARCS  model  based  on  the  frequencies  of  “Mostly  True”  and  “Very  True”  that  were 
selected  for  each  component.  Additionally,  the  qualitative  data  suggested  a considerable  amount  of  students  did 
not  perceive  using  the  tutorial  as  a satisfactory  experience.  Therefore  the  objective,  based  on  the  pre-  analysis , 
was  to  increase  students’  satisfaction  level  towards  the  tutorial  with  a feasible  instructional  intervention. 


TABLE  2.  Comparison  of  frequencies  of  "Very  True  ” selected  by  participants  among  ARCS  components 


Pre -Motivational  Survey 

Attention 

Relevance 

Confidence 

Satisfaction 

Frequency  of  response  indicating 
“Mostly  True’’  and  “Very  True” 

2692 

3568 

2749 

711 

combined 

Identifying  Motivational  Strategies 

The  qualitative  data  analysis  from  the  pre-motivational  survey  indicated  that  the  main  issue  influencing 
students  ’ motivation  was  the  transfer  between  learning  programming  syntax  and  using  the  syntax  in  problem- 
solving scenarios.  The  problem  of  transferability  is  common  in  abstract  cognitive  skill  instruction  such  as 
programming  or  mathematics.  Quilici  and  Mayer  (2002)  conducted  a study  on  the  transferability  of  statistical 
skills  to  word  problems , which  suggests  the  transfer  of  knowledge  can  be  facilitated  by  providing  systematic 
and  frequent  training.  Renkl  et  al.  (2002)  also  suggests  that  by  providing  problem-solving  examples  with 
gradually  increasing  levels  of  difficulty  students  were  able  to  develop  complete  problem-solving  skills  by 
themselves.  The  low  ratings  on  the  Satisfaction  component  provided  further  support  for  designing 
corresponding  motivational  strategies.  In  order  to  enhance  the  Satisfaction  level,  students  should  have  more 
opportunities  to  receive  feedback  and  have  feelings  of  accomplishment  by  exercising  newly -learned  knowledge 
(Keller,  1987b).  As  a result,  a student -made  glossary  was  developed  (Table  3),  which  allowed  students  to  define 
and  practice  MATLAB  syntax  prior  to  being  given  engineering  problems  to  solve. 


TABLE  3.  Example  of  student-made  glossary 


MATLAB 

Syntax 

Overview  of  Syntax 

Test  Case(s) 

Expected  Result 
for  Each  Test 
Case 

MATLAB  Command  Used 
for  Each  Test  Case  and 
Result 

who 

Description:  Displays 
a list  of  all  the 
variables  available  in 
the  workspace. 

Input  Arguments: 

none 

General  Format: 

» who 

Enter  these 
MATLAB 
commands: 
»C  = 0 
» x = 3 
» who 

Lists  the 
variables  C and  x 
(and  anything 
else  previously 
created)  as  being 
available  in  the 
workspace 

As  given  under  Test  Cases. 

Result: 

Your  variables  are: 

C x 

save 

Description:  Saves 
all  results,  or  specified 
results,  to  a MAT  file 
Input  Arguments: 
filename  and  specific 
variables  (optional) 

General  Format: 

»save  filename 
»save  filename  varl 

1.  Create  three 
variables. 

2.  Save  all  of  the 
variables  you've 
created  in  a 
filename  called 
vars_all. 

3.  Save  two  of 
the  three 
variables  you 

Saves  variables 
to  a MAT  file 

Where  are  the 
files  saved?  Give 
the  exact  path 
name. 

To  the  current 
directory  which 
should  be  h:/Mv 

X = 1 
Y = 2 
Z = 3 

save  vars_all 
save  vars_two  X Y 
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MATLAB 

Syntax 

Overview  of  Syntax 

Test  Case(s) 

Expected  Result 
for  Each  Test 
Case 

MATLAB  Command  Used 
for  Each  Test  Case  and 
Result 

var2 ... 

created  in  a 
filename  called 
vars_two. 

Documents/m- 

tutor 

Implementation  of  Motivational  Strategy:  The  Student-Made  Glossary 

The  student-made  glossary  was  implemented  as  a homework  assignment  towards  the  end  of  the 
semester.  The  glossary  items  came  from  the  last  M -Tutor  learning  unit  assigned  and  included  syntax  needed  to 
complete  problem-solving  assignments  for  the  remainder  of  the  semester.  Students  were  asked  to  fill  out  all  four 
columns  as  the  first  part  of  the  assignment.  Later  the  commands  seen  in  the  glossary  were  applied  to  other 
homework  problems.  The  columns  conform  to  Renkl’s  idea  (Renkl,  et  ah,  2002)  of  increasing  difficulty.  In  the 
first  column,  the  student  describes  the  programming  syntax  in  terms  of  how  and  why  to  use  it.  In  the  next 
column,  the  student  must  generate  a small  sample  case  using  the  syntax.  In  the  next  two  columns,  the  student 
generates  an  answer  without  the  computer,  then  feeds  their  sample  code  into  the  MATLAB  program  and 
records  its  response. 

Post-Motivational  Survey  Data  Analysis 

The  same  instrument  (IMMS)  used  in  the  pre  -motivational  survey  was  administrated  to  conduct  the 
post-motivational  survey  to  make  a valid  comparison,  as  shown  in  Table  4. 

A one-way  ANOVA  was  applied  to  data  analysis  to  investigate  the  significance  of  variance  between 
pre  - and  post-motivational  surveys  as  the  result  of  implementing  the  motivational  intervention,  that  is,  the 
student -made  glossary. 

The  analysis  indicated  that  the  use  of  the  glossary  significantly  lowered  students’  satisfaction  as  shown 
in  Table  4 based  on  the  aforementioned  frequency  method,  which  contradicts  our  hypothesis  that  the 
satisfaction  level  would  increase  when  students  were  provided  with  opportunities  to  practice  newly  acquired 
skills  and  to  gain  meaningful  feedback.  Possible  reasons  contributing  to  the  result  will  be  addressed  in  the 
Discussion  section. 


TABLE  4.  One-way  ANOVA  on  students ' satisfaction  towards  M-Tutor  between  Pre-  and  Post -motivational 
surveys 


Pre  Mean 

Post  Mean 

F 

Significance  level 

N=  957 

0.75 

0.57 

10.717 

0.001 

Post-Motivational  Survey  and  Glossary  Survey 

The  purpose  of  implementing  the  glossary  survey  (Cronbach’s  Alpha=  0.926)  was  to  collect  data  on 
students’  reactions  to  using  it  as  a supplementary  learning  tool  and  its  relation  to  students  ’ satisfaction  level  on 
the  post-motivational  survey.  To  thoroughly  evaluate  the  implemented  motivational  strategy  (i.e.  student-made 
glossary),  it  is  important  to  measure  students  ’ initial  reaction  towards  the  glossary  as  the  first  level  of  evaluation 
(Kirkpatrick,  1998).  Interestingly  the  linear  regression  analysis  suggests  there  is  a significant  relationship 
between  students’  reactions  towards  the  glossary  and  the  post-satisfaction  response  frequencies  as  shown  in 
Table  5.  Students’  reactions  towards  the  glossary,  which  was  developed  based  on  the  different  levels  of 
intellectual  skill  hierarchical  analysis,  had  a statistically  significant  relationship  with  students’  post-satisfaction 
level  as  defined  by  Keller’s  ARCS  Model. 

TABLE  5.  Linear  regression  analysis  between  students’  reaction  towards  glossary  and  Post-satisfaction 
responses  frequencies 


Y=aX+b 

Coefficient 

t statistic 

P value 

R Square 

a(  Glossary  Reaction) 

0.1103 

9.2985 

9.42E-20 

0.083 

The  study  suggests  that  the  cognitively  based  instructional  intervention  (i.e.  student -made  glossary)  is 
influential  on  the  variance  of  students’  Satisfaction  level  in  a computer-based  instructional  setting,  though  the 
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effect  is  considered  negative  towards  students’  overall  motivational  gain.  Further  discussion  will  explore  the 
underlying  reasons  as  to  how  the  construction  of  their  problem-solving  schema  affects  students’  perceived 
motivational  levels  during  the  development  of  higher  cognitive  skills. 

As  the  result  of  evaluation  for  the  implemented  intervention,  the  significant  linear  regression  relationship 
between  students’  positive  reaction  to  glossary  and  their  post-satisfaction  level  suggests,  interestingly,  the 
possibility  of  other  factors  affecting  the  motivational  gain  induced  by  the  glossary,  which  also  will  be  discussed 
in  the  next  section. 


Discussion 

Traditionally  motivational  studies  have  not  been  conducted  very  systematically,  perhaps  due  to  the 
complex  nature  of  motivation  and  the  difficulty  in  measuring  it.  This  study  uses  a generic  systematic  model, 
ADD1E,  often  used  in  instructional  design,  with  the  following  steps:  analysis,  design,  development, 
implementation,  and  evaluation.  The  ADD1E  model  is  general  enough  to  approach  most  problems,  including 
motivation.  The  appearance  of  each  component  of  ADD1E  in  most  of  the  accepted  instructional  design  models 
today  is  a sign  of  its  theoretical  grounding. 

This  study  focused  primarily  on  the  Satisfaction  component  after  the  pre- motivational  survey,  and 
showed  significant  change  after  the  intervention.  Students  felt  less  satisfied  towards  M-Tutor  after  the  use  of  the 
glossary.  However,  the  regressive  relationship  between  students’  positive  reaction  to  the  glossary  (i.e.  how  they 
like  the  intervention)  and  their  post-satisfaction  level  suggests  otherwise.  Before  we  delve  into  the  reasons  as  to 
why  the  loss  of  motivational  gain,  it  is  necessary  to  first  to  understand  the  function  of  student -made  glossary 
and  how  it  is  compatible  with  Keller’s  definition  of  the  Satisfaction  component.  Keller  breaks  Satisfaction  up 
into  three  parts:  natural  consequences,  positive  consequences,  and  equity  (Keller,  1987). 

It  could  be  that  either  positive  consequences  or  equity  relate  to  this  study.  The  design  and  development 
of  student-made  glossary,  however,  focused  on  natural  consequences  mainly  because  it  can  give  students  a 
meaningful  way  to  transfer  the  information  they  have  learned.  The  student-made  glossary  is  a cognitive  advance 
organizer  that  addresses  that  aspect  of  satisfaction  by  helping  student  create  a glossary  of  terms,  leading  them 
from  knowing  what  the  term  is  to  practicing  how  to  use  it.  The  organizer  categorizes  students’  knowledge,  then 
provides  a way  for  students  to  use  it  in  a meaningful  way  in  context  and  therefore  meeting  the  "natural 
consequences"  requirement  of  satisfaction. 

Given  the  pedagogically -sound  theoretical  foundation  and  rationale  of  designing  and  developing  the 
student -made  glossary  derived  from  either  cognitive  loading  theory  (e.g.  worked  example)  (Sweller,  1994)  or 
the  ARCS  motivational  design  model  (Keller,  1987a, b),  the  negative  effect  on  motivational  gain  induced  by  the 
glossary  raises  issues  not  only  associated  with  the  developmental  portion  of  instructional  intervention,  but 
redirects  our  attention  to  the  overall  research  design,  especially  in  the  implementation  part  of  the  study. 

The  pre -motivational  survey  was  administered  in  the  first  quarter  of  the  semester  while  the 
instructional  intervention  (i.e.  student -made  glossary)  was  not  in  place  until  the  last  quarter  due  to  time  needed 
for  the  pre -motivational  survey  data  analysis  and  the  design  and  development  of  the  glossary.  This  temporal  gap 
is  typical  in  the  developmental  research  (Richey&  Nelson,  1996).  Additionally  the  study  was  conducted 
simultaneously  with  the  progression  of  the  course.  Therefore  it  is  very  likely  that  the  glossary  is  not  the  cause  of 
the  motivational  loss.  Various  factors  such  as  course  structure,  complexity  of  learning  tasks,  interaction  between 
teaching  staff  (i.e.  professors  and  teaching  assistants)  and  students , and  the  utilization  of  M-Tutor  could  have 
influenced  students  ’ motivation  levels.  It  is  questionable  whether  the  student-made  glossary  could  overcome  the 
interactions  among  the  aforementioned  factors  in  terms  of  students  ’ motivational  gain  or  loss.  In  order  to  better 
measure  any  instructional  intervention’s  effectiveness  especially  on  perceived  motivation,  a more  controlled 
study  setting  (e.g.  smaller  sample  size)  with  a much  shorter  time  gap  between  pre  - and  post-motivational 
surveys  needs  to  be  applied. 


Conclusion 

This  study  presents  the  concept  and  feasibility  of  using  a generic  instructional  design  process  (ADDIE 
model)  to  motivational  evaluation.  It  also  demonstrates  the  use  of  the  ARCS  Model  of  Motivational  Design  for 
analysis  and  evaluation  of  students’  learning  motivation  when  using  an  existing  computer-based  tutorial. 
Triangulation  of  quantitative  and  qualitative  data  provided  insight  on  the  impact  of  instructional  components  on 
students’  learning  motivation  as  represented  by  four  motivational  components , which  is  also  helpful  for  future 
revision  of  the  computer-based  tutorial.  Finally  the  issue  of  effective  implementation  of  an  instructional 
intervention  for  motivational  studies  poses  important  considerations  when  migrating  from  the  realmof 
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developmental  research  to  amore  replicable  research  design. 
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Abstract 

Given  the  emerging  trend  of  innovating  Instructional  Design  process  (ID)  for  Human  Performance 
Technology  (HPT)  related  application  in  order  to  more  effectively  addressing  organizational  performance 
problems  in  industries , it  is  necessary  to  prepare  students  majoring  in  Instructional  Design  or  Educational 
Technology  in  terms  of  how  they  can  apply  and  further  develop  the  concepts  and  skills  learned  from  the  field  of 
ID  to  a more  complicated  and  challenging  HPT  setting  for  the  increasing  demand  seen  in  job  markets.  This 
paper  presents  a case  study  on  the  design,  development,  and  implementation  of  a graduate  level  HPT  course  for 
the  purpose  of  better  developing  ID  students  for  future  HPT  related  tasks.  Issues  such  as  (1)  locating  potential 
HPT  client  in  an  academic  setting,  (2)  the  dynamic  interaction  among  stakeholders,  (3)  Tools  used  to  facilitate 
the  communication  (i.e.  visual  modeling  method,  WebCT),  and  (4)  effectives  of  project-base  learning  for  HPT 
courses  are  discussed. 


What  is  HPT 

Application  of  HPT  in  Education  Settings 

Human  Performance  Technology  (HPT)  is  defined  as  “a  systematic  approach  to  improving 
productivity  and  competence,  through  a process  of  analysis,  intervention  selection  and  design,  development, 
implementation  and  evaluation,  designed  to  of  programs  to  most-cost-effectively  influence  human  behavior  and 
accomplishment”  (ISPI,  2000).  It  is  usually  applied  in  business  or  industrial  settings  to  address  performance 
problems , but  is  increasingly  applied  in  public  sectors  and  community  settings  (Schaffer  & Richardson,  2004). 
The  International  Society  for  Performance  Improvement  also  provides  a job  description  for  a human 
performance  technologist:  ‘‘HP  technologist’s  are  those  who  adopt  a systems  view  of  performance  gaps, 
systematically  analyze  both  gap  and  system,  and  design  cost-effective  and  efficient  interventions  that  are  based 
on  analysis  data,  scientific  knowledge,  and  documented  procedures”  (Stolovich  & Keeps,  1992). 

The  infrastructural  knowledge  (i.e.  organizational  behavior,  employer  and  employee  relationship, 
incentives  for  better  organizational  performance)  for  facilitating  the  solution  design  decision-making  process  for 
HPT  is  also  more  complicated  and  dynamic  in  nature  as  compared  to  an  instructional  problem.  Though  the 
concept  of  developing  value-added  and  cost-effective  non-instructional  interventions  (Reiser  & Dempsey,  2002) 
to  solve  HPT  problems  is  somewhat  novel  for  many  instructional  designers,  it  is  clear  to  see  the  compatibility 
between  the  task  of  instructional  designer  and  human  performance  technologist  in  terms  of  using  a systematic 
and  analytical  approach  to  solve  performance  problems . While  both  ISD  and  HPT  use  a similarproblem-solving 
approach,  the  level  of  organizational  impact  and  the  measurability  of  results  that  is  a hallmark  of  HPT  requires  a 
much  stronger  emphasis  on  analysis. 

Given  that  there  is  an  increasing  need  for  human  performance  solutions  in  either  for-profit  industries  or 
educational  settings,  the  venues  for  developing  capable  human  performance  technologist  are  in  demand. 

Actually  many  existing  graduate  level  instructional  technology,  instructional  design,  and  educational 
technology  programs  have  begun  to  establish  a solid  foundation  to  support  the  growing  demand  for 
performance  improvement  education. 

This  paper  presents  a case  study  on  the  design,  development,  and  implementation  of  a graduate  level 
HPT  course  in  order  to  share  the  instructor’s  as  well  as  students’  experiences.  The  process  and  tools  employed 
to  conduct  the  human  performance  project  are  also  discussed  in  order  to  establish  instructionally  practical  and 
pragmatic  guidelines  for  others  interested  in  creating  similar  learning  experiences  for  students. 
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Process 

The  first  challenge  for  the  instructor  was  to  introduce  the  fundamental  knowledge,  from  concept 
definitions  to  theory  and  research  of  HPT,  to  novice  graduate  students  in  a constrained  time  frame.  It  was 
critical  to  provide  the  students  with  both  the  broad  view  in  a wide  range  of  topics  on  HPT,  and  the  specific 
systems  and  models  to  prepare  them  for  the  solutions  of  real  HPT  problems.  This  was  achieved  following  a 
step-by-step  learning  process  through  which  students  were  guided  to:  1)  compare  their  personal  definitions  of 
the  field  against  the  textbook  and  supplementary  readings;  2)  create  a system  model  and  describe  its 
interdependence  with  other  systems,  3)  review  articles  from  Performance  Improvement  Quarterly  or 
Performance  Improvement  to  identify  the  theoretical  framework  and  discuss  the  research  support  for  the 
performance  intervention;  and  4)  compare  and  contrast  three  different  HPT  process  models.  After  they  gained 
enough  understanding  on  the  topic,  students  completed  an  Individual  Performance  Analysis  Report  in  which 
they  integrated  the  concept,  theory  and  system  models  to  address  a problem  in  which  they  were  interested.  It 
also  served  as  practice  in  using  analysis  tools  for  their  mo  re  complex  HPT  project  in  the  next  phase  of  the 
course. 

HPT  Performance  Analysis  Process 

The  second  challenge  was  to  locate  projects  that  not  only  could  provide  practical  experiences  for 
students,  but  were  also  achievable  within  the  semester  timeframe.  Since  students  were  novice  to  the  HPT  filed, 
the  instructor  took  the  initiative  to  select  the  project  topics  and  group  the  students  accordingly.  At  the  beginning 
of  the  course,  the  instructor  invited  and  introduced  the  “clients”,  the  on-campus  Information  Technology  (IT) 
division  in -charge  persons  to  the  class,  to  help  students  better  understand  the  expectations  and  identify  the 
possible  problem  scopes.  Once  the  general  problem  areas  for  projects  were  identified,  the  instructor’s  role 
became  more  like  a consultant,  providing  necessary  guidance  to  the  students  who  actually  took  the 
responsibility  to  apply  HPT  to  address  real  world  problems. 

Tools 

Turing  Research  into  Results — a Guide  to  Selecting  the  Right  Performance  Solutions  (Clark  & Estes, 
2002)  and  Performance  Consulting  (Robinson,  & Robinson,  1995)  were  selected  as  the  textbooks  for  this 
course.  Robinson,  & Robinson’s  (1995)  Performance  Model  and  Performance  Relationship  Map  offered  a good 
starting  point  for  students  to  take  a systems  view  of  identified  performance  problems.  Students  collaborated  in 
developing  their  own  system  models  and  relationship  maps  for  different  performance  problems,  which  lay  the 
foundation  for  later  cause  analysis  and  intervention  selection. 

The  instructor  also  helped  students  to  focus  the  analysis  on  the  top  three  causes  of  performance 
problem:  Knowledge  and  Skill,  Motivation,  and  Organizational  Causes  (Clark  & Estes,  2002).  In  the  case  of 
one  identified  problem,  P2P  file  sharing,  illegal  file  sharing  was  more  likely  to  be  caused  by  motivation  and 
organizational  culture  than  a lack  of  knowledge  or  skill . In  addition,  the  instructor  introduced  the  Performance 
Pyramid  (Wedman  & Graham,  1998),  which  visually  laid  out  the  possible  performance  problem  causes  into 
building  blocks.  It  provided  a particularly  useful  instrument  to  ground  the  design  and  development  of  the 
investigation  questionnaire,  a major  tool  for  the  project  teams  to  collect  information. 

Project  Management 

Communication  Tools  Both  project  teams  used  private  discussbn  forums  in  WebCT  to  archive  team 
files,  chat  rooms,  and  the  university  web  mail  to  communicate  with  clients,  the  instructor  and  one  another. 
Effective  communication  was  important  to  keep  the  project  on  track  and  the  effort  focused  in  the  right  areas. 
WebCT  discussion  was  used  among  the  instructor  and  the  students  to  clarify  questions  and  concerns,  generate 
ideas,  explore  solutions  and  offer  in  -time  assistance.  E-mails  were  frequently  utilized  to  exchange  information, 
update  progress  and  report  results.  Finally  the  online  survey  results  were  put  into  MS  Excel  spread  sheets  and 
visual  charts  were  created  to  present  and  interpret  data  to  the  clients.  In  doing  so  students  not  only  learned  how 
to  apply  HPT  theory  and  principles  into  specific  context,  but  also  practiced  consulting  skills  such  as  how  to  “ask 
the  right  question  right”  (Robinson,  & Robinson,  1995)  and  how  to  visualized  the  presentation  of  collected  data 
to  help  the  clients  reach  conclusions. 

Client  Involvement  _ As  in  any  successful  HPT  project,  it  was  critical  to  involve  the  client  throughout 
the  process  of  performance  analysis.  In  both  projects,  the  students  conducted  structured  meetings  with  the 
clients  before,  during  and  after  the  performance  analysis  to  define  the  problem,  report  progress,  and  obtain 
feedback.  At  each  crucial  stage  of  the  performance  analysis,  the  students  would  acquire  approval  from  the 
clients  before  taking  further  actions.  The  client -oriented  approach  helped  to  build  solid  support  for  the  HPT 
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project  as  well  as  ensure  customer  satisfaction  and  buy-in  to  the  process  and  results. 


P-Based  Learning:  Project,  Process,  People,  Problem 

This  course  was  offered  from  a graduate  level  Educational  Technology  program  at  Purdue  University. 

A ”P-based”  approach  to  course  development  was  used  including  project,  problem,  process,  and  people-based 
methods.  Project  selection  for  this  course  was  mainly  conducted  by  the  instructor  due  to  students’  lack  of 
experience  and  contacts  on  campus.  The  most  time  consuming  part  of  project  selection  process  was  to  explain 
the  concept  of  human  performance  to  potential  clients. 

The  first  key  requirement  for  project  selection  is  that  the  gap  between  current  state  of  identified 
performance  and  desired  state  of  performance  was  quantifiable  in  terms  of  dollar  value,  hours  of  manpower 
consumed,  numbers  of  filed  customer  complaints,  etc.  In  certain  cases  especially  within  educational  settings  the 
measurable  outcome  initially  cannot  easily  be  identified,  for  example,  attitude  changes . Therefore  the  clients, 
course  instructor,  and  students  have  to  work  closely  to  identify  projects  with  measurable  outcome.  The  second 
concern  is  the  constrained  timeframe  and  resources  of  the  course  (i.e.  eight  weeks  from  project  initiation  and 
final  reporting).  The  scope  and  depth  of  projects  both  were  considered  in  relation  to  students’  novice  level  of 
analysis  and  problem-solving  skills  within  the  HPT  context. 

Although  it  is  not  fair  to  use  a dichotomous  method  to  categorize  all  human  performance  problems  into 
inductive  and  deductive  ones,  this  approach  did  help  students  in  the  course  to  initiate  their  problem-solving 
process.  Two  topics  were  selected  from  an  on-campus  Information  Technology  division.  One  project  focused  on 
student’s  awareness  of  available  information  security  (IS)  resources  provided  by  university.  The  other  focused 
on  student’s  awareness  of  the  detrimental  consequences  of  using  the  campus  network  to  illegally  download 
copyrighted  files  via  Peer-to-Peer  (P2P)  file  sharing. 

The  objective  of  IS  project  was  to  identify  student  behaviors  related  to  securing  his  or  her  computer 
with  virus  protection  software,  for  example;  whereas  the  P2P  project  was  geared  toward  trying  to  understand 
why  students  use  the  campus  network  to  execute  illegal  transactions.  The  IS  project  eventually  converged  all 
data  into  a set  of  awareness  procedures  that  could  be  implemented  immediately  while  the  data  from  P2P  project 
produced  a number  questions  requiring  further  investigation. 

It  was  obvious  in  the  initial  stage  of  project  development  that  each  team  (IS  and  P2P)  should  approach 
their  specific  problem  with  different  mindsets.  The  IS  team  was  looking  for  an  incremental,  relatively  short  term 
intervention.  The  P2P  was  developing  a data-driven  rationale  for  further  investigation  of  the  problem  as  the 
result  of  HPT  process.  Students  in  this  course  were  fortunate  enough  to  encounter  both  types  of  HPT  problems 
(i.e.  inductive  and  deductive)  in  one  semester  and  consequently  adjusted  certain  analysis  process  (e.g.  gap 
analysis,  performance  map  analysis)  in  order  to  accommodate  project  objectives. 

Conclusion  and  Implication 

The  HPT  course  discussed  in  this  paper  offered  students  inductive  and  deductive  types  of  problems  to 
solve  over  the  course  of  the  semester.  By  continuing  to  dissect  and  learn  about  the  many  facets  of  human 
performance  theory  it  was  clear  that  many  things  other  than  knowledge  and  skill  cause  poor  performance.  The 
ability  to  quickly  apply  theory  provided  students  with  the  opportunity  to  facilitate  problem  solving  in  a complex 
setting  with  real  time  lines.  In  the  future,  students  will  be  introduced  to  HPT  frameworks  and  processes  in  much 
the  same  way  as  in  the  current  course.  However,  future  courses  will  allow  students  to  select  their  own  projects 
rather  than  rely  only  on  instructor-selected  projects.  Due  to  the  amount  of  work  and  time  needed  to  complete  the 
semester-long  project,  students  in  current  and  future  classes  will  be  given  a longer  timeline  for  completion. 
Additional  online  discussions  will  be  available  to  support  learners  through  peer  feedback,  and  a web-based 
performance  support  tool  will  be  used  to  provide  student  analysts  with  process  support,  tools,  and  report 
templates. 

A significant  change  in  the  course  is  conducted  is  the  introduction  of  methods  to  promote  reflective 
inquiry.  Learners  reflect  upon  each  of  the  major  analysis  processes:  define,  analyze  and  select,  by  writing  brief 
statements  about  their  experiences  while  completing  processes.  Learners  also  self-assess  their  confidence  in 
completing  processes  by  rating  the  quality  of  each  deliverable.  It  is  hoped  that  using  action  research  methods  to 
explore  the  processes  of  novice  analysts  will  support  deeper  learning  and  better  understanding  of  the  analysis 
process  in  general. 


Sidebar:  One  Student’s  Application  of  the  HPT  Model 

HPT  is  a meaningful  model  for  a technology  coordinator.  With  the  emphasis  of  the  job  on  the  staff 
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development  of  K-12  teachers,  results  showed  that  the  training  the  teachers  completed  was  not  being 
implemented  into  their  classrooms.  Having  completed  the  course  has  provided  a foundation  of  the  HPT  to 
analyze  this  situation.  The  model  will  be  incorporated  in  current  and  future  plans  of  the  school  district.  For 
example,  performance  concerns  that  exist  were  being  fixed  by  issuing  additional  training.  Of  course,  this  was 
the  incorrect  method  of  dealing  with  the  issues  because  performance  was  not  modified. 

As  a result  before  trainings  are  organized,  performance  is  looked  upon  as  what  is  occurring  compared 
to  the  ideal  performance.  Beginning  from  this  point  has  brought  about  change  and  performance  improvement. 
Issues  of  expectations  and  feedback  as  well  as  desire  to  perform  have  been  identified.  This  has  provided  a 
starting  point  for  improvement. 
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Abstract 

Team  efficacy  is  a group-level  motivational  construct,  which  is  the  extension  of  self-efficacy.  It  has 
been  argued  that  teams  with  higher  team  efficacy  perform  more  efficiently  than  the  teams  with  lower  team 
efficacy.  The  purpose  of  this  study  is  to  examine  the  relationship  between  team  efficacy  and  team  performance 
in  Personnel  Qualification  Standards  (PQS)  book  development  and  revision  teams  in  the  U.S.  Navy.  The  results 
of  the  study  will  be  reported  and  discussed. 


Introduction 

Recent  years  have  seen  an  increasingly  widespread  use  of  teams  in  organizations.  Many  organizations 
rely  on  teams  to  accomplish  important  goals.  However,  teams  are  not  always  effective.  Some  teams  perform 
better  than  the  others.  It  is  suggested  that  processes  that  occur  within  a team  may  help  account  for  real 
differences  in  team  effectiveness  (Brannick,  et  al.,  1997).  Several  team  process  variables  have  been  investigated 
in  relation  to  team  performance,  such  as  team  cooperation  (Mathieu,  et  ah,  2000),  team  coordination  (Stout  et 
ah,  1990;  Entin  & Serfaty,  1999),  and  team  communication  (Mathieu  et  ah,  2000).  For  some  time,  there  has 
been  a fair  amount  of  research  on  these  team  variables.  However,  only  a few  studies  investigated  the 
relationship  between  the  motivational  process  of  team  efficacy  (collective  efficacy)  and  team  performance. 

Team  efficacy  (Bandura  1982)  is  a motivational  group-level  construct  that  originates  with  Bandura’s 
concept  of  self-efficacy  (1977).  It  refers  to  “a  group’s  shared  belief  in  its  conjoint  capabilities  to  organize  and 
execute  the  courses  of  action  required  to  produce  given  levels  of  attainment”  (Bandura,  1997,  p477).  A couple 
of  studies  have  reported  positive  results  regarding  the  relationship  between  team  efficacy  and  team  performance 
(Gibson,  1999,  Knight  et  al,  2001,  Little  et  ah  1997;  Spink,  1992).  Nevertheless,  most  of  these  studies  are 
conducted  with  short-term  artificial  teams  in  the  laboratory  settings  (Gibson,  1999;  Knight  et  al,  2001),  with 
sports  teams  (Spink,  1992),  or  manufacturing  work  teams  (Little  et  ah,  1997).  Few  studies  have  been  conducted 
to  examine  the  relationship  between  team  efficacy  and  team  performance  with  performance  improvement  teams 
in  areal  work  setting. 

The  purpose  of  this  study  is  to  examine  the  relationship  between  team  efficacy  and  team  performance 
in  Personnel  Qualification  Standards  (PQS)  development  teams  in  the  U.S.  Navy.  More  specifically,  it  is 
intended  to  investigate  if  teams  with  higher  team  efficacy  perform  better  than  teams  with  lower  team  efficacy. 

Method 

Participants 

The  participants  of  the  study  were  Navy  personnel  and  civilians  who  worked  in  teams  to  revise  U.S. 
Navy  PQS  books.  32  participants  in  four  teams  participated  in  this  study.  Each  of  them  worked  in  teams  during 
a few-day-long  workshop  to  discuss  and  revise  a particular  PQS  book.  The  teams  were  formed  between  8 and  2 
weeks  before  the  workshop.  Some  team  members  had  already  known  each  other  before  attending  the  workshop, 
while  others  had  no  prior  contact.  Each  team  consisted  of  4 to  17  members  (comprised  of  one  facilitator,  one 
model  manager,  and  the  remaining  were  subject  matter  eiperts  (SME)).  During  the  workshops,  SMEs  discussed 
and  made  suggestions  on  book  revision,  while  the  role  of  the  facilitators  was  to  manage  the  discussion  process, 
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record  the  changes  to  be  made,  and  keep  the  team  on  track.  The  model  manager  was  also  a SME,  but  had  an 
overall  assignment  of  keeping  the  books  current.  The  workshops  lasted  from  2 to  4 days. 

Material 

A PQS  contains  minimum  knowledge  and  skills  that  navy  personnel  must  demonstrate  in  order  to 
qualify  to  stand  watches  or  perform  other  specific  routine  duties  necessary  for  the  safety,  security,  or  proper 
operation  of  a ship,  aircraft,  or  support  system.  The  book  presents  performance  requirements  that  the  Navy 
personnel  are  required  to  demonstrate  a reasonable  amount  of  knowledge  comprehension  or  skill  proficiency. 
All  PQS  books  consisted  of  three  sections:  100  section  (Fundamentals)  contained  the  fundamental  knowledge 
necessary  to  understand  the  watchstation/workstation  books,  200  section  (Systems)  contained  the  information 
on  the  systems  that  will  be  required  to  operate  at  watchstation/workstation,  300  section  (Watchstations) 
contained  the  tasks  that  will  be  required  to  satisfactorily  perform  in  order  to  achieve  final  PQS  qualification  for 
a particular  watchstation/workstation. 

Measures 

Team  efficacy 

Team  efficacy  was  measured  with  a 14-item  questionnaire  developed  by  the  researchers.  It  is  a 10- 
point  Likert  scale  questionnaire,  with  0 representing  ‘not  confident  at  all’  and  10  representing  ‘extremely 
confident’ . The  instrument  items  were  identified  based  on  a task  analysis  conducted  with  a panel  of  3 subject 
matter  experts.  The  instrument  asked  the  participants  to  indicate  the  degree  of  confidence  in  his/her  team’s 
ability  in  performing  tasks  involved  in  the  PQS  book  revision  process  (e.g.  Your  confidence  (0-10)  in  your 
team 's  ability  to  verify  references  in  the  PQS  book).  Each  team  member  completed  the  questionnaire 
independently.  Then,  each  individual’s  rating  of  each  question  was  aggregated  and  the  mean  rating  was 
calculated  indicating  the  rating  of  team  efficacy.  Bandura  (2000)  claims  the  aggregation  method  is  useful  for 
teams  of  high  interdependence. 

Team  performance 

Team  performance  was  measured  by  the  number  of  revisions  made  to  the  PQS  book  per  hour.  As  the 
study  was  conducted  in  a real  setting,  we  had  no  control  of  the  amount  of  time  each  team  would  spend  on  the 
tasks.  The  number  of  revisions  per  hour  should  be  a reasonable  measure  of  performance  across  the  teams. 

Procedure 

Immediately  before  each  team  started  the  workshop  on  the  first  day,  the  workshop  observer  gave  a 
copy  of  the  team  efficacy  questionnaire  to  each  participant.  It  was  stressed  that  each  participant  read  each 
question  carefully  and  that  for  each  particular  task  listed,  he/she  needed  to  rate  his/her  confidence  level  on 
his/her  team’s  confidence  in  performing  the  task.  After  each  participant  independently  completed  the 
questionnaire,  copies  were  collected.  Then  the  observer  observed  the  workshop  process  and  took  field  notes 
including  the  time  each  team  started  discussing  a revision,  the  process  of  the  discussion,  and  the  time  that  the 
team  reached  a consensus  and  completed  the  revision. 

The  same  procedures  were  repeated  for  each  team  on  the  last  day  of  the  workshop  when  the  second- 
time questionnaire  was  administered  to  the  participants.  Team  efficacy  was  measured  twice  because  previous 
research  indicated  that  efficacy  beliefs  formed  before  task  performance  can  be  different  from  those  formed 
afterward  (Gibson,  1999).  We  chose  to  measure  the  construct  twice  to  see  their  relationship  to  team 
performance  respectively. 

Data  Analysis 

For  each  team,  the  original  PQS  book  before  the  workshop  (Start  Book)  and  the  final  PQS  book  after 
the  workshop  (End  Book)  were  collected  for  comparison.  One  researcher  compared  the  Start  Book  and  End 
Book  for  all  the  four  teams  following  a coding  system  that  was  developed  by  the  researchers.  The  coding 
system  was  developed  based  on  researchers’  experiences  with  previous  PQS  workshops  as  well  as  observations 
of  all  the  four  workshops.  Every  single  change  in  the  books  was  coded  using  notations  in  the  coding  system. 

The  parts  that  remained  intact  were  also  marked.  After  all  the  books  were  coded,  the  revisions  made  to  each 
PQS  book  were  categorized  and  the  total  number  of  revisions  was  counted.  In  addition,  the  observer’s  field 
notes  were  analyzed  and  the  total  number  of  hours  spent  on  the  book  revision  process  for  each  team  was 
counted.  The  number  of  total  revisions  divided  by  the  number  of  total  hours  spent  on  the  revision  process  was 
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then  calculated  as  the  performance  measure  of  the  teams. 

Pearson’s  r correlation  was  computed  to  determine  the  relationship  between  team  efficacy  measured  at 
two  times  (at  the  beginning  of  the  workshop  and  at  the  end  of  the  workshop)  and  team  performance. 

Results  and  Discussion 

The  purpose  of  the  study  is  to  seek  the  relationship  between  team  efficacy  and  team  performance. 
Pearson’s  r correlation  showed  that  team-efficacy  as  measured  both  at  Time  1 and  Time  2 was  positively 
correlated  to  team  performance,  but  the  results  were  not  significant  (Time  1:  r=.14,  Time  2:  r=.03).  Table  1 
displays  the  correlation  data. 


Table  1:  Correlations  between  team  efficacy  and  team  performance 


Team-efficacy  (Time 
1) 

Team-efficacy 
(Time  2) 

Performance 

Team-efficacy  (Time  1) 

1 

- 

Team-efficacy  (Time  2) 

.478** 

1 

- 

Performance 

.14 

.03 

1 

Note:  **Correlation  is  significant  at  0.01  level  (two-tailed) 


One  limitation  of  the  study  is  the  small  sample  size.  A larger  sample  size  needs  to  be  gathered  in  order 
to  more  accurately  measure  the  relationship  between  team-efficacy  and  team-performance.  Another  issue  of 
concern  is  that  the  teams  studied  were  of  different  sizes.  For  example,  one  team  consisted  of  only  four 
members;  while  another  team  consisted  of  fourteen  members.  Team  size  might  have  played  a role  in  team 
efficacy  and  performance.  These  two  issues  of  concern  may  have  influenced  the  results  of  the  study.  However, 
the  strength  of  the  research  is  that  it  studied  real  performance  improvement  teams,  which  is  rarely  available  in 
the  current  literature. 
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Abstract 

This  presentation  will  focus  on  the  use  of  a custom  developed  Innovation  Component  Configuration 
Map  (ICCM)  to  measure  technology  integration  practices  of  faculty  in  Schools,  Colleges,  and  Departments  of 
Education  (SCDEs).  This  study  investigated  the  relationship  between  the  level  of  technology  integration  fidelity 
(high,  moderate  or  low)  by  SCDE  faculty  and  a)  access  to  adec/uate  support  from  technological  infrastructure, 
b)  access  to  adequate  support  from  human  infrastructure,  and  c)  personal  attitude  toward  computer  use. 

Introduction 

The  primary  goal  of  this  study  was  to  explore  and  identify  best  practices  in  technology  integration  by 
higher  education  faculty,  specifically  higher  education  faculty  in  Schools,  Colleges,  and  Departments  of 
Education  (SCDEs).  This  research  study  grew  out  of  a perceived  need  to  identify  and  contribute  to  the  sparse 
knowledge  base  in  current  literature  related  to  concrete  classroom  practices  in  technology  integration  by  higher 
education  faculty  in  SCDEs  (Tharp,  1997;  Willis,  Thompson  & Sadera,  1999).  This  population  is  of  particular 
concern  given  the  inherent  responsibility  and  expectation  for  these  faculty  to  model  effective  technology 
integration  within  their  instruction  in  the  preparation  of  preservice  teachers. 

Theoretical  Background 

The  literature  documents  that  the  preparation  for  preservice  teachers  in  the  area  of  technology 
integration  is  inadequate  (e.g.,  Garcia,  1998;  Hannafin,  1999;  Moursund  & Bielefeldt,  1999;  Poole,  1998; 
Schrum,  1999).  There  are  isolated  examples  of  excellence  in  technology  integration  (e.g.,  Eakin,  1997;  Michael, 
1998;  Persichitte,  Caffarella  & Tharp,  1999;  Studler  & Wetzel,  1999).  Much  of  the  research  literature  in 
technology  integration  is  limited  to  use  of  technology  rather  than  on  integration  of  technology  in  education. 

Also  very  little  of  the  technology  integration  literature  focuses  on  higher  education  faculty.  Previous  research 
(Persichitte  et  al.,  1999)  documents  the  importance  of  variables  that  influence  technology  integration: 
technological  infrastructure,  human  support  infrastructure,  and  attitude  toward  computer  use.  The  work  of 
change  theorists  and  diffusion  of  innovation  scholars  such  as  Rogers  (1995),  Hall  and  Hord  (2001),  and  Fullan 
(1993)  offers  a sound  theoretical  foundation  for  further  research  in  the  area  of  technology  integration.  However, 
the  diffusion  rate  of  technology  integration  practices  among  SCDE  faculty  is  very  low  and  hence,  the  focal 
point  of  this  research  was  an  investigation  of  the  process  of  change  and  diffusion  of  instructional  technology 
among  teacher  education  faculty.  The  literature  also  suggests  that  although  many  K-12  schools  and  higher 
education  settings  have  established  benchmarks  or  standards  for  the  integration  of  technology  into  classrooms, 
no  model  or  methodology  exists  for  substantiating  technology  standards  with  actual  classroom  practices  (Mills, 
2001). 


Research  Questions 

The  following  research  questions  guided  this  study. 

RQ  1.  To  what  extent  do  SCDE  faculty  report  examples  of  technology  integration  that  parallel 
examples  of  best  practices  in  the  current  literature? 
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RQ  2.  To  what  extent  do  SCDE  faculty  report  high  fidelity,  moderate  fidelity,  or  low  fidelity 
technology  integration  practices  in  their  teaching? 

RQ  3.  Is  there  a relationship  between  level  of  technology  integration  fidelity  (high,  moderate  or 
low)  by  SCDE  faculty  and  access  to  adequate  support  from  technological  infrastructure? 

RQ  4.  Is  there  a relationship  between  level  of  technology  integration  fidelity  (high,  moderate  or 
low)  by  SCDE  faculty  and  access  to  adequate  support  from  human  infrastructure? 

RQ  5.  Is  there  a relationship  between  level  of  technology  integration  fidelity  (high,  moderate  or 
low)  by  SCDE  faculty  and  their  personal  attitude  toward  computer  use? 

Methodology 

Instruments 

The  variables  of  primary  interest  in  this  research  study  were:  technology  integration  practices, 
implementation  fidelity  of  technology  integration  by  higher  education  faculty,  attitudes  of  faculty  toward 
computers,  and  technological  and  human  infrastructure  in  support  of  technology  integration.  Technology 
integration  practices  and  implementation  fidelity  by  faculty  were  measured  by  an  Innovation  Component 
Configuration  Map  (ICCM)  that  was  custom  developed,  field  tested,  and  reviewed  by  experts  in  the  area  of 
technology  integration.  The  Attitudes  Toward  Computer  Usage  Scale  (ATCUS)  (Popovich,  Hyde,  Zakrajsek,  & 
Blumer,  1987)  was  utilized  to  collect  data  related  to  computer  attitudes  of  SCDE  faculty  participants. 
Information  about  technological  and  human  infrastructure  was  obtained  from  open  ended  and  multiple  choice 
questions  included  as  a part  of  the  Demographic  Questionnaire. 

Participants  and  Data  Collection  Procedures 

The  focus  of  this  research  was  higher  education  faculty  members  associated  with  SCDEs.  Hence  the 
representative  population  selected  for  this  study  was  individual  members  of  the  American  Association  of 
Colleges  of  Teacher  Education  (AACTE).  AACTE  is  the  principal  professional  association  for  college  and 
university  leaders  with  responsibility  for  educator  preparation.  The  AACTE  2002  membership  list  included 
5,323  individual  members.  From  the  individual  membership,  600  faculty  were  randomly  sampled  and  asked  to 
participate  in  this  study. 

To  answer  the  research  questions  addressed  in  this  study,  600  instrumentation  packets  (ICCM, 

ATCUS,  and  Demographics)  were  disseminated  to  members  of  the  American  Association  of  Colleges  of 
Teacher  Education  (AACTE),  via  a multi- mode  (paper-based,  online  and  e-mail)  method.  After  two  follow-up 
reminders,  completed  surveys  were  received  from  208  participants  resulting  in  a response  rate  of  36%. 
Interestingly,  of  the  208  participants,  53  responded  to  the  online  version,  154  via  the  paper-based  version,  and 
only  one  participant  responded  via  the  e-mail  attachment.  Of  the  AACTE  members  returning  completed 
surveys,  56.7%  were  female,  91.4%  were  Assistant,  Associate,  and  Professors  with  89.9  % holding  a doctorate 
degree,  99%  reported  full  time  employment  status,  and  95.8%  were  affiliated  with  a college  or  a university.  The 
age  group  of  participant  faculty  ranged  from  20  to  72  years  (M  = 50.74,  SD  = 11.57)  with  teaching  experience 
ranging  from  0 to  48  years  (M  = 25,  SD  = 8.96). 


Results 

Research  question  1 focused  on  the  technology  integration  practices  of  higher  education  faculty  for 
which  the  ICCM  served  as  the  primary  data  collection  instrument.  Results  of  the  analysis  indicate  that,  on  an 
average,  79.1%  of  the  SCDE  faculty  participants  were  close  to  demonstrating  best  practices  in  technology 
integration,  and  that  this  mean  for  ICCM  scores  was  significantly  different  from  the  best  practice  score  of  75. 
The  responses  on  the  ICCM  were  fairly  normally  distributed  with  a large  variance  (M  = 79.1,  SD  = 19.51). 

Research  question  2 explored  the  classification  of  the  SCDE  faculty  into  three  distinct  groups  of 
technology  integration  fidelity  levels.  The  ICCM  allowed  for  the  categorization  into  three  distinct  groups, 
resulting  in  56.7%  in  the  high  fidelity,  38%  in  the  moderate  fidelity,  and  5.3%  in  the  low  fidelity  group.  The 
chi-squared  goodness  of  fit  test  determined  that  there  were  significant  differences  in  the  proportions  of  the  three 
fidelity  groups  favoring  high  fidelity. 

Research  questions  3,  4,  and  5 explored  the  relationships  between  six  predictor  variables  (four  factors 
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of  the  ATCUS,  technological  and  human  support  available  to  faculty)  and  technology  integration  fidelity  levels 
(dependent  variable).  A multinomial  logistic  regression  analysis  predicted  the  relationship  among  the  set  of 
predictor  variables  and  the  dependent  variables  to  be  significant.  Slightly  over  58%  of  the  participants  were 
correctly  classified  by  fidelity.  However,  ATCUS  factor  2 (Positive  Reactions  to  Computers)  and  ATCUS 
factor  4 (Comfort  with  Familiar  Computer-Related  Mechanisms)  were  the  only  individual  significant  factors  in 
differentiating  between  high  and  low  fidelity  groups.  ATCUS  factor  4 (Comfort  with  Familiar  Computer- 
Related  Mechanisms)  was  the  only  significant  factor  that  differentiated  between  moderate  and  high  fidelity 
groups. 

The  results  of  an  ANOVA  indicated  a statistically  significant  difference  between  faculty  who  reported 
adequate  technological  infrastructure  and  faculty  who  reported  inadequate  technological  infrastructure.  There 
was  also  a significant  interaction  between  human  infrastructure  support  and  computer  anxiety.  Faculty  with 
high  computer  anxiety  improved  on  their  technology  integration  skills  when  provided  with  adequate  support 
from  human  infrastructure. 


Conclusion 

This  research  study  investigated  the  complex  relationships  among  these  constructs:  technological 
infrastructure,  human  infrastructure,  attitudes  toward  computers,  and  technology  integration  fidelity  with  a 
random  sample  of  higher  education  faculty  representing  SCDEs.  This  study  contributes  to  the  technology 
integration  literature  in  the  following  ways:  ( 1)  provides  evidence  of  tangible  classroom  practices  in  technology 
integration  for  teaching  and  learning  in  higher  education,  (2)  offers  a contemporary  ICCM  to  measure 
technology  integration  among  higher  education  faculty,  (3)  provides  an  updated  description  of  faculty 
characteristics  for  SCDE  faculty  who  integrate  technology  in  their  teaching,  and  (4)  supports  Roger’s  (1995) 
theoretical  framework  and  the  change  literature  associated  with  the  Concerns-Based  Adoption  Model  (CBAM) 
(Hall  & Hord,  2001). 

Note:  The  detailed  instruments  (ICCM,  ATCUS  and  the  Demographic  Questionnaire)  used  in  this  study  would 
be  provided  to  the  conference  participants  at  the  presentation. 
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Abstract 

This  study  examined  how  differences  in  communication  style  affect  how  likely  particular  types  of  messages 
(e.g.  arguments,  evidence,  critiques,  explanations ) were  able  to  elicit  critical  responses  during  four  online 
debates.  Event  sequence  analysis  was  used  to  compare  the  response  probabilities  for  each  type  of  message 
across  messages  that  used  expository  versus  epistolary  styles  of  communication  observed  in  the  four 
asynchronous  threaded  discussions.  The  results  suggests  that  when  a message  is  posted  to  challenge  an 
opposing  viewpoint,  that  message  is  significantly  less  likely  to  elicit  a return  response  from  the  opposition  when 
the  message  acknowledges  individuals  by  name  or  presents  a direct  reference  to  an  individual’ s preceding 
statements.  A more  detailed  and  exploratory  analysis  of  the  interactions  revealed  that  this  style  of 
communication  might  have  contributed  to  a decrease  in  the  frequency  of  evidence  and  subsequent  discussion  of 
supporting  explanations  needed  to  defend  the  challenged  viewpoints  and  arguments. 

Introduction 

Computer- mediated  communication  (CMC)  provides  opportunities  for  learner-to-learner  interaction 
and  enables  learning  communities  to  engage  in  more  reflective  critical  thinking.  Argumentation  is  one  of  the 
fundamental  collaborative  inquiry -based  learning  strategies  for  increasing  critical  thinking  skills  in  online 
settings.  These  skills  involve  the  processes  of  building  arguments  to  support  a position,  and  considering  and 
weighing  evidence  and  counter-evidence  in  developing  supporting  arguments.  Innovative  approaches  to 
facilitating  student  participation  in  collaborative  argumentation  have  been  developed  in  computer-supported 
collaborative  learning  (CSCL)  and  computer-supported  collaborative  argumentation  (CSCA).  One  approach  to 
scaffolding  and  facilitating  collaborative  argumentation  is  to  structure  the  discussions  such  that  constraints  are 
placed  on  when  and  what  types  of  functional  moves  and  messages  can  be  posted  to  discussions  (Cho  & 

Jonassen,  2002;  Duffy  et  ah,  1998;  Jonassen  & Remidez,  2001;  McAlister,  2001).  The  other  approach 
incorporates  intelligent  systems  and  pedagogical  agents  that  use  formalized  models  of  argumentation  to  actively 
diagnose  students'  performance  and  suggest  immediate  courses  of  action  (Eleuterio,  Barthes  & Bartolozzi, 

2002;  Jacques,  Oliveira,  & Vicari,  2002a;  Jacques  et.  Al,  2002b;  Karacapilidis  & Papadias,  2001). 

Despite  these  technical  advances  to  support  the  functions  students  perform  during  online  discussions 
and  regardless  of  what  technology  is  used  to  facilitate  discussion,  a growing  body  of  research  now  suggests  that 
student  participation  in  online  discussions  can  be  influenced  by  the  communication  styles  of  its  participants. 
When  and  how  often  a student  responds  to  a message  may  depend  not  just  on  what  is  said  in  the  message  (in 
terms  of  function  of  the  message),  but  also  by  how  the  message  was  delivered  in  terms  of  communication  style. 
Significant  differences  in  communication  styles  have  been  found  between  males  and  females.  For  example,  men 
tend  to  assert  opinions  strongly  as  facts,  place  more  value  on  presenting  information  using  an  expository  style, 
are  more  likely  to  use  crude  language,  violate  online  rules  of  conduct,  engage  in  more  adversarial  exchanges, 
and  terminate  exchanges  when  there  are  disagreements  (Fahy,  2002;  Herring,  1993;  Savicki  et  al.,  1996).  In 
contrast,  females  are  more  likely  to  qualify  and  justify  their  assertions,  use  expressions  that  convey  more 
epistolary  roles,  make  apologies,  and  in  general,  manifest  a more  consensus-making  orientation  and  epistolary 
style.  Furthermore,  females  are  more  upset  by  violations  of  politeness  and  are  more  likely  to  challenge 
participants  that  violate  rules  of  conduct  and  (Smith,  McLaughlin  & Osborne,  1997).  Although  each  gender  may 
have  a tendency  towards  one  communication  style,  groups  have  also  been  found  to  modify  their  communication 
styles  in  the  direction  of  the  majority  gender  (Baym,  1996;  Herring,  1996). 
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However,  the  impact  of  communication  styles  on  when  and  how  often  messages  are  able  to  elicit 
responses  that  generate  more  critical  and  more  substantive  discussions  has  yet  to  be  examined  and  empirically 
investigated.  The  lack  of  empirical  research  can  be  attributed  to  the  absence  of  appropriate  theories,  methods 
and  tools  capable  of  operationalizing  and  producing  precise  measurements,  descriptions  and  assessments  of 
group  interactions  and  processes  in  online  discussions  (Fahy,  2001  & 2002;  Garrison,  2000).  The  main 
problems  lie  in  the  coding  and  analysis  of  computer  conference  messages  (Rourke,  Anderson,  Garrison,  Archer, 
2001)  and  the  sequential  nature  of  messages  and  responses  (Gunawardena,  Lowe  & Anderson,  1997).  The 
difficulty  in  coding  message  content  is  in  establishing  the  unit  of  analysis  because  messages  often  contain 
multiple  ideas  that  serve  multiple  functions.  As  a result,  the  contents  of  a message  must  be  classified  into 
multiple  codes,  making  it  nearly  impossible  to  map  message-response  sequences  in  terms  of  pre -defined 
message  categories  (Levin,  Kim  & Riel,  1990;  Newman,  Johnson,  Cochrane  & Webb,  1996;  Gunawardena, 
Lowe  & Anderson,  1997).  These  methodological  problems  have  prevented  researchers  from  conducting  a closer 
and  more  detailed  examination  of  the  relationships  between  messages  and  responses  - particularly  the 
functional,  temporal,  social  and  semantic  relationships  between  exchanged  messages  and  responses  (Jeong, 
2004a). 

One  solution  to  this  problem  is  to  require  students  to  pre  -classify  their  contributions  to  discussions 
using  a pre -determined  set  of  message/response  categories.  This  constrains  each  message  to  serve  only  one 
function  at  a time,  and  establishes  a clear  unit  of  analysis.  This  approach  has  been  used  in  a number  of 
computer-supported  collaborative  argumentation  (CSCA)  systems  (Cho  & Jonassen,  2002;  Duffy  et  ah,  1998; 
Jonassen  & Remidez,  2001;  McAlister,  2001).  The  ACT  system  (Sloffer,  Dueber  & Duffy,  1999),  for  example, 
is  a threaded  discussion  board  that  is  designed  to  scaffold  online  debates  by  requiring  students  to  pre -classify 
each  posting  to  one  of  six  response  categories  - proposal,  counter-proposal,  supporting  reasons,  detracting 
reasons,  supporting  evidence  and  detracting  evidence.  McAlister  (2003)  proposed  a synchronous  chat  tool  to 
support  collaborative  argumentation  by  requiring  students  to  pre -classify  messages  to  inform,  question, 
challenge,  reason,  support  or  maintain  chat  discussions.  Within  each  of  these  response  categories,  students  are 
able  to  choose  a specific  sentence  opener  (e.g.  "A  counter-argument  is...")  to  channel  students'  thoughts  by  the 
process  of  completing  the  sentence  in  a way  that  fits  with  the  opener. 

Purpose 

Using  the  methods  described  above,  the  purpose  of  this  study  was  to  determine  the  precise 
probabilities  in  which  messages  with  a given  function  (e.g.  arguments,  supporting  evidence,  critiques  and 
explanations)  and  communication  style  (epistolary  vs.  expository)  are  able  to  successfully  elicit  responses  that 
contribute  to  discussions  that  critically  examine  claims  and  alternative  viewpoints.  The  research  questions  that 
were  specifically  addressed  in  this  study  were  the  following: 

1.  Level  of  Interaction  Between  Opposing  Viewpoints.  When  conducting  discussions  to  critically 
examine  alternative  viewpoints,  how  does  the  use  of  epistolary  styles  of  communication  affect  how 
likely  students  will  respond  back  to  students  who  challenge  their  claims  with  counter-arguments  and 
supporting  explanations? 

2.  Impact  on  Level  of  Critical  Discussion.  What  are  the  implications  of  using  versus  not  using  these 
specific  styles  of  communication  on  the  level  and  depth  of  discussion  across  all  presented  arguments 
and  claims? 


Method 

Participants 

The  participants  (n  = 17)  were  graduate  students  from  a major  university  in  the  Southeast  region  of  the 
U.S.,  consisting  of  1 1 female  and  6 male  students,  and  ranging  from  20  to  40  years  in  age.  The  students  were 
enrolled  in  a 16 -week  online  graduate  introductory  course  to  distance  learning  required  to  complete  a Master's 
program  in  instructional  systems  with  a major  in  distance  learning.  The  gender  of  each  student  was  determined 
at  the  beginning  of  the  online  course  through  personal  introductions  and  posted  biographies. 

Debate  procedures 

Students  were  required  to  participate  in  a series  of  four  highly  structured  debates  using  threaded 
discussion  forums  in  Blackboard,  a web-based  course  management  system.  The  discussions  were  highly 
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structured  given  that:  a)  student  participation  in  the  debates  and  other  scheduled  discussions  throughout  the 
course  contributed  to  20%  of  the  course  grade;  b)  for  each  debate,  students  were  required  to  post  at  least  four 
messages  to  receive  full  credit  for  each  debate;  c)  prior  to  each  debate,  students  were  randomly  assigned  to  one 
of  two  teams  (but  balanced  by  gender)  to  either  support  or  oppose  a given  position;  and  d)  students  were 
required  to  vote  on  the  team  that  presented  the  strongest  arguments  after  each  debate. 

The  purpose  of  each  debate  was  to  critically  examine  design  issues,  concepts  and  principle  in  distance 
learning  covered  during  the  week  of  the  debate.  For  example,  students  debated  over  the  following  claims:  “The 
Dick  & Carey  ISD  model  is  an  effective  model  for  designing  the  instructional  materials  for  this  course”,  “The 
role  of  the  instructor  should  change  when  teaching  at  a distance”,  “Type  of  media  does  not  make  any  significant 
contributions  to  student  learning”,  and  “Given  the  data  and  needs  assessment,  the  fictitious  country  of  NED 
should  not  develop  a distance  learning  system”.  Students  were  instructed  to  support  and  refute  presented  claims 
and  viewpoints  with  arguments,  evidence,  explanations,  and  critiques. 

Online  debate  messages  and  message  labels 

Students  were  presented  a list  of  four  response  categories  during  the  debates  designed  to  scaffold 
argumentation.  The  response  categories  were  based  loosely  on  Toulmin’s  (1958)  model  of  argumentation.  The 
response  categories  and  their  definitions  were  presented  to  students  prior  to  participating  in  the  debates.  Each 
student  was  required  to  classify  each  posted  message  by  response  category  by  inserting  the  corresponding  label 
into  the  subject  headings  of  every  message.  Students  were  required  to  limit  the  content  of  their  messages  to 
address  one  and  only  one  response  category  at  a time.  The  experimenter  occasionally  checked  the  message 
labels  to  determine  if  students  had  appropriately  labeled  their  messages  according  to  the  described  procedures. 
No  participation  points  were  awarded  for  a given  debate  if  a student  failed  to  properly  label  one  or  more  posted 
messages.  However,  students  were  able  to  return  to  a message  at  any  time  to  correct  for  errors  in  their  labels. 


Symbol 

Description  of  symbol 

+ 

Identifies  a messaae  costed  bv  a student  assianed  to  the  team  sucoortina  the 
given  claim/statement 

- 

Identifies  a messaae  costed  bv  a student  assianed  to  the  team  oooosinathe 
given  claim/statement 

ARG# 

Identifies  a messaae  that  oresents  one  and  onlv  one  araument  or  reason  for 
using  or  not  using  chats  (instead  of  threaded  discussion  forums).  Number  each 
posted  argument  by  counting  the  number  of  arguments  already  presented  by 
your  team.  Sub-arguments  need  not  be  numbered.  ARG  = "argument". 

EXPL 

Identifies  a reply/message  that  provides  additional  support,  explanation, 
clarification,  elaboration  of  an  argument  or  challenge. 

BUT 

Identifies  a reply/message  that  questions  or  challenges  the  merits,  logic, 
relevancy,  validity,  accuracy  or  plausibility  of  a presented  argument  (ARG)  or 
challenge  (BUT). 

EVID 

Identifies  a reply/message  that  provides  proof  or  evidence  to  establish  the 
validity  of  an  argument  or  challenge. 

Figure  1 - Example  instructions  on  how  to  label  messages  during  the  online  debates 

The  purpose  of  labeling  and  constraining  each  message  to  a specific  function  was  to  make  the  inter- 
relationships between  students’  contributions  explicit  and  to  enable  students  to  visualize  the  structure  of  their 
arguments  (Jeong  & Juong,  2004b).  The  second  reason  for  using  these  procedures  is  that  they  enabled  the 
experimenter  to  clearly  establish  each  message  as  a unit  of  analysis.  With  each  message  assigned  to  a specific 
response  category,  the  types  of  message-response  sequences  observed  in  the  debates  could  be  clearly  identified 
and  counted  to  determine  their  relative  frequencies.  Previous  studies  on  group  interaction  in  CMC  did  not 
succeed  in  measuring  message-response  sequences  (Gunawardena,  Lowe,  & Anderson,  1997;  Newman, 
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Johnson,  Cochrane,  & Webb,  1996)  because  students’  messages  and  responses  often  addressed  multiple 
functions  or  response  categories  at  the  same  time.  As  a result,  mapping  the  relationships  between  messages  and 
responses  was  a difficult  if  not  impossible  task.  In  this  study,  however,  the  use  of  student-labeled  messages  was 
found  to  be  an  adequate  solution  to  resolving  some  of  the  previous  problems  by  establishing  a clear  unit  of 
analysis.  Although  these  procedures  appear  at  first  sight  to  be  artificial  and  perhaps  intrusive,  this  method  has 
been  implemented  in  a number  of  computer-supported  collaborative  argumentation  (CSCA)  systems  for 
scaffolding  argumentation  and  problem  solving  (Carr  & Anderson,  2001;  Cho  & Jonassen,  2002;  McAlister, 
2003;  Sloffer,  Dueber,  & Duffy,  1999;  Veerman,  Andriessen,  & Kanselaar,  1999). 

In  addition,  students  were  also  instructed  to  identify  each  mess^e  by  team  membership  by  adding  an 
for  opposing  or  a “+”  for  supporting  team  at  the  end  of  each  label  (e.g.  +ARG,  -ARG).  These  tags  allowed  the 
students  to  easily  locate  the  exchanges  between  the  opposing  and  supporting  teams  within  the  discussion 
threads  (e.g.  +ARG->-BUT)  and  respond  to  the  exchanges  to  advance  or  defend  their  team’s  position.  An 
example  discussion  thread  from  a debate  is  illustrated  in  Figure  2. 

Figure  2.  Example  of  online  debate  with  labeled  messages 


r 

E SUPPORT  statement  because... 

Student  names 

Sat  Oct  2,  2004  11:18  am 

r 

B +ARG#1  MedialsButAMereVehicle 

Student  names 

Mon  Oct  4,  2004  8:47  pm 

r 

E-EVID  MedialsButAMereVeh... 

Student  names 

Tue  Oct  5,  2004  7:09  pm 

r 

E +But  RelativitvTheorv... 

Student  names 

Tue  Oct  5,  2004  9:43  pm 

r 

-But  RelativitvThe. . . 

Student  names 

Sat  Oct  9,  2004  10:12  am 

r 

-BUT  Whataboutemotions? 

Student  names 

Tue  Oct  5,  2004  9:53  pm 

r 

+EVID  DistEdEffectiveAsF2F 

Student  names 

Tue  Oct  5,  2004  10:40  pm 

r 

-BUTMediaamerevehicle 

Student  names 

Wed  Oct  6,  2004  8:19  pm 

r 

E +EVID  MooreConcurs 

Student  names 

Wed  Oct  6,  2004  10:07  pm 

r 

+EXPLMediaSelectionCo... 

Student  names 

Sun  Oct  10,  2004  12:35  am 

r 

B -BUT  WellChosenEffect. . . 

Student  names 

Sun  Oct  10,  2004  4:31  pm 

r 

+But  SunnortinaRes... 

Student  names 

Sun  Oct  10,  2004  5:37  pm 

r 

B -BUTMediaismorethenamere... 

Student  names 

Fri  Oct  8,  2004  5:30  pm 

r 

+BUT  SuoDortinaEviden... 

Student  names 

Sat  Oct  9,  2004  8:51  am 

r 

-BUT  LearninaNotSimolvAP... 

Student  names 

Mon  Oct  1 1 , 2004  9:54  am 

r 

B +ARG2  Standards  for  teachina 

Student  names 

Wed  Oct  6,  2004  1:48  pm 

r 

+But  Clarification? 

Student  names 

Sun  Oct  10,  2004  5:39  pm 

r 

B +ARG3  MediaUnrelatedtoLearn... 

Student  names 

Wed  Oct  6,  2004  3:12  pm 

r 

B -BUTMediaUnrelatedtoLear... 

Student  names 

Wed  Oct  6,  2004  8:26  pm 

r 

B +BUT  MediaSelection 

Student  names 

Thu  Oct  7,  2004  9:20  am 

r 

-BUT  MediaSelection 

Student  names 

Sun  Oct  10,2004  11:21  am 

r 

+EVID  MethodNotMedia 

Student  names 

Wed  Oct  6,  2004  1 1 :04  pm 

r 

B-BUT  MediaUnrelatedtoLea... 

Student  names 

Sat  Oct  9,  2004  10:59  am 

Data  analysis 

Inter-rater  reliability.  To  determine  the  extent  in  which  the  messages  were  labeled  correctly  by  students, 
two  of  the  four  debates  were  randomly  selected  and  coded  by  the  experimenter  to  determine  inter-rater 
reliability  (work  in  progress). 

Data  preparation  prior  to  analysis.  To  prepare  the  data  for  analysis,  computer  software  was  written  by  the 
experimenter  to  download,  tabulate  and  compile  the  student -labeled  messages  from  the  Blackboard  discussion 
forums  into  Microsoft  Excel.  The  experimenter  reviewed  all  the  messages  to  identify  and  tag  any  messages  that 
contained  expressions  that  reflected  more  conversational  and  epistolary  exchanges.  The  expressions  that  were 
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found  to  exhibit  the  epistolary  style  were  direct  references  to  previous  statements  (e.g.  “I  agree  when  you  say 
that...),  and  references  to  individual’s  names  (e.g.  “Hi  Bob”).  All  messages  that  contained  these  expressions 
were  tagged  with  an  “s”  (e.g.  BUTs).  As  a result,  a total  of  eight  message  categories  (ARG,  ARGs,  BUT,  BUTs, 
EVID,  EVIDs,  EXPL,  EXPLs)  were  used  to  code  the  messages. 

Event  sequence  analysis.  Event  sequence  analysis  (Bakeman  & Gottman,  1997)  was  used  to  identify  and 
quantify  the  observed  interaction  patterns  between  messages  that  exhibited  different  communication  styles. 
Sometimes  referred  to  as  lag  analysis  or  Markovian  chain  analysis,  the  purpose  of  event  sequence  analysis  is  to 
determine:  a)  the  probability  in  which  a given  event  is  able  to  elicit  one  or  more  subsequent  events;  and  b)  the 
probabilities  in  which  specific  types  of  events  are  likely  to  follow  a given  event.  Event  sequence  analysis  has 
effectively  used  in  communications  research  to  study,  for  example,  communication  patterns  in  the  conversations 
and  interactions  between  married  couples  (Bakeman  & Gottman,  1997  pp.  184-193;  Gottman,  1979),  children  at 
play  (Bakeman  & Brownlee,  1982),  mother  and  infant  at  play  (Stern,  1974),  and  humans  and  computer- 
interfaces  (Olson,  Herbsleb,  & Rueter,  1994). 

A computer  program  (Jeong,  2003),  the  Discussion  Analysis  Tool  (DAT),  was  developed  by  the 
experimenter  to  perform  the  event  sequence  analysis.  DAT  tallied  the  frequency  of  each  message-response 
interaction.  The  response  frequencies  for  each  target  message  were  then  converted  into  relative  frequencies,  or 
transitional  probabilities  (Bakeman  & Gottman;  1997).  The  transitional  probabilities  measured  how  likely  a 
particular  type  of  response  was  posted  in  reply  to  a particular  type  of  message  relative  to  all  the  other  types  of 
responses  elicited  by  the  given  message. 

Transitional  state  diagrams.  The  observed  transitional  probabilities  between  messages  with  different 
communication  styles  and  function  were  converted  into  transitional  state  diagrams  using  DAT.  In  the  diagrams, 
each  response  category  was  represented  in  a node.  Directional  arrows  were  drawn  from  one  node  to  another 
node  to  represent  the  relative  frequency  of  each  observed  message-response  interaction.  The  density  of  the 
directional  arrows  connecting  the  nodes  illustrates  the  strengths  of  the  transitional  probabilities  between  the 
nodes.  This  graphical  representation  of  the  interactions  provided  a means  to  readily  identify  prevalent  patterns 
of  interaction  produced  by  messages  that  exhibited  expository  styles  and  epistolary  style.  Most  of  all,  the 
diagrams  provided  a visual  approach  to  identifying  differences  in  message-response  sequences  between  the 
message  types. 


Results 

Summary  statistics  and  overall  level  of  interaction.  Figure  3 shows  the  frequency  matrix  that  summarizes 
the  number  of  observed  messages  and  responses  across  categories.  The  four  debates  produced  a total  of  323 
messages.  Of  the  323  messages,  239  were  posted  in  reply  to  a previous  message  and  166  of  the  messages  did 
not  elicit  any  replies.  The  matrix  also  shows  how  many  messages  of  a given  category  were  observed,  and  how 
many  responses  were  elicited  by  the  message  within  a given  category.  For  example,  the  debates  generated  a 
total  of  84  arguments  (ARG),  65  messages  (BUT)  that  challenged  its  merits  with  out  using  an  epistolary  style 
versus  20  challenging  messages  (BUTs)  with  epistolary  style.  In  Figure  4 is  the  transitional  probability  matrix, 
which  shows  the  relative  frequency  of  each  type  of  response  elicited  by  each  message  category.  For  example, 
50%  of  responses  to  ARG  were  BUT  (65  of  the  129  replies)  versus  16%  BUTs  (20  of  the  129  replies).  This 
matrix  also  shows  the  overall  response  rates  for  each  message  category.  For  example,  81%  of  the  arguments 
elicited  a response  based  on  the  finding  that  only  16  of  the  84  did  not  receive  any  responses  at  all.  The  overall 
response  rate  across  all  message  categories  was  67%. 
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Figure  3 - Matrix  with  message  and  response  frequencies 
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Figure  4 - Matrix  with  transitional  probabilities  between  messages  and  responses 
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Level  of  interaction  between  opposing  viewpoints.  By  analyzing  only  the  interactions  produced  by 
exchanges  between  participants  on  opposing  teams  (e.g.  +ARG^  -BUT,  -BUT->+BUT,  -BUT->+EVID,  etc.), 
the  results  showed  that  messages  presented  with  personal  acknowledgments  and  direct  references  to  an 
individual’s  previous  statements  (epistolary  style)  received  significantly  fewer  return  responses  (or  rebuttals) 
from  the  opposition  (X  = .39,  STD  = .60,  n =67)  compared  to  messages  that  were  presented  without  using  this 
style  (X  = .61,  STD  = .83,  n = 256),  t(321)  = 2.08,  p = .038,  with  the  effect  size  of  -.31. 

Impact  on  level  of  critical  discussion.  To  explore  the  potential  impact  of  using  the  epistolary  style  of 
communication  on  the  subsequent  direction  and  level  of  critical  discussion,  the  transitional  probabilities  from 
the  upper-left  and  lower-right  quadrants  in  the  probability  matrix  (Figure  4)  were  examined  separately  to 
compare  the  types  of  discussion  that  would  or  could  be  produced  if  all  messages  in  a discussion  were  presented 
with  versus  without  the  epistolary  style.  The  transitional  probabilities  produced  by  messages  without  the 
epistolary  style  (upper  right  quadrant)  are  depicted  in  the  left  diagram  in  Figure  5.  The  diagram  on  the  right 
depicts  the  interactions  produced  only  by  messages  presented  with  the  epistolary  style. 
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Figure  5 - Possible  patterns  of  interaction  when  discussions  consist  only  of  expository  messages  versus  only 
epistolary  messages. 


Expository  ->  Expository  Epistolary  ->  Epistolary 

message-response  exchanges  message-response  exchanges 

A comparison  of  the  two  diagrams  reveals  possible  differences  in  outcomes  following  interactions 
produced  with  epistolary  style  versus  without  epistolary  style  of  communication.  When  the  epistolary  style  was 
not  used,  challenges  (BUT)  directed  at  an  argument  (ARG->BUT)  were  more  likely  to  elicit  responses  with 
evidence  in  defense  of  the  challenged  argument.  In  other  words,  the  sequence  of  events  produced  without  the 
epistolary  style  (ARG-^BUT^EVID)  was  more  likely  to  occur  (in  15%  of  replies  to  BUT)  when  the  epistolary 
style  was  not  used  to  challenge  an  argument.  When  the  epistolary  style  was  used  to  present  a challenge 
(ARG^BUTs),  this  interaction  was  less  likely  to  lead  to  the  production  of  evidence  (in  only  8%  of  replies  to 
BUTs).  The  left  diagram  in  Figure  5 also  suggests  that  once  evidence  was  elicited  by  a challenge  (without  the 
epistolary  style),  students  were  more  likely  to  sustain  and  develop  the  discussion  with  explanations  (ARG 
BUT  EVID  ->  EXPL  ->  EXPL).  These  differences  suggest  that  the  use  of  expository  style  produced 
discussions  that  exhibited  more  reflection  and  critical  examination  of  arguments  during  the  online  debates. 
However,  these  observations  must  be  considered  with  caution  because  these  observations  are  based  on  low 
frequencies  of  messages  presented  with  the  epistolary  style. 

Discussion 

The  findings  of  this  study  show  that  the  epistolary  style  of  communication  - namely  the  direct  reference  to 
individuals  by  name  and  direct  reference  to  previous  statements  - inhibited  the  critical  discussion  and 
examination  of  arguments,  and  that  critical  discussion  is  more  likely  to  occur  when  students  adopt  and 
implement  a more  expository  rather  than  epistolary  style  of  discourse  during  computer-supported  collaborative 
argumentation.  This  finding  may  appear  to  contradict  the  prediction  that  a more  epistolary  or  conversational 
style  of  communication  encourages  individual  participants  (particularly  female  participants)  to  reciprocate 
responses  and  extend  a conversation  by  helping  to  build  rapport  between  participants  (Fahy,  2002).  However, 
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the  findings  in  this  study  showed  that  the  epistolary  style  inhibited  rather  than  supported  student  responses  to 
challenges.  One  possible  explanation  for  this  unexpected  finding  is  that  the  confrontational  nature  of 
collaborative  argumentation  may  have  been  intensified  or  even  personalized  when  students  made  direct 
reference  to  individual  names  and  previous  statements.  Engaging  in  discussion  without  this  epistolary  style 
would  seem  to  have  helped  the  students  maintain  focus  on  the  ideas  and  not  the  personalities  involved  in  the 
discussions,  thereby  reducing  the  sense  of  contentiousness  inherent  in  the  debate  activity. 

The  findings  from  the  exploratory  analysis  also  suggest  that  the  use  of  the  epistolary  style  could  have  a 
potentially  negative  effect  on  the  level  of  reflection,  critical  analysis  and  discussion.  The  analysis  of  the  state 
diagrams  revealed  patterns  to  suggest  that  the  epistolary  style  can  reduce  the  frequency  of  evidence  and 
explanations  presented  in  defense  of  arguments  following  challenges  to  the  arguments.  To  thoroughly  test  the 
effects  of  discourse  styles  on  students’  interaction  patterns  and  the  outcomes  produced  by  the  interactions, 
future  studies  will  (a)  need  to  collect  a larger  corpus  of  data  to  gather  sufficient  number  of  messages  with 
contrasting  styles,  or  (b)  conduct  a controlled  experiment  in  which  students  in  multiple  groups  are  instructed  to 
implement  different  styles  of  discourse  as  they  participate  in  online  collaborative  activities  such  as  debates, 
problem-solving,  and  case-studies. 

In  conclusion,  this  study  was  successful  in  making  a first  attempt  at  determining  the  effects  of  discourse 
styles  on  student  performance  in  CMC  when  no  previous  studies  have  succeeded  or  attempted  to  determine  its 
actual  impact.  The  findings  in  this  study  serve  to  demonstrate  the  efficacy  of  using  event  sequence  analysis, 
combined  with  the  use  of  response  constraints  to  label  messages  by  function,  to  precisely  measure  and  study  the 
effects  of  epistolary  versus  expository  styles  in  computer-mediated  communication.  Ultimately,  this  study  will 
also  serve  as  a model  for  investigating  the  effects  of  other  styles  and  protocols  commonly  observed  in  online 
communication,  which  include  the  use  of  emoticons,  qualifiers,  humor,  rhetorical  questions,  and  the  explicit 
versus  discrete  expression  of  disagreements.  Measuring  the  precise  effects  of  these  communication  styles  and 
protocols  will  enable  researchers  and  instructors  to  develop  more  refined  strategies  and  computational  models 
for  supporting  and  optimizing  student  performance  in  computer-supported  collaborative  learning. 

References 

Bakeman,  R.  & Brownlee,  J.  (1982).  Social  rules  governing  object  conflicts  in  toddlers  and  preschoolers.  In  K. 
H.  Rubin  & H.  S.  Ross  (Eds.),  Peer  relationships  and  social  skills  in  childhood (pp.  99- 1 1 1).  New 
York:  Springer- Verlag. 

Bakeman,  R.,  & Gottman,  J.  (1997).  Observing  Interaction:  An  introduction  to  sequential  analysis,  Cambridge, 
University  Press. 

Baym,  N.  (1996).  Agreements  and  disagreements  in  a computer- mediated  discussion.  Research  on  Language 
and  Social  interaction , 29(4),  315-345. 

Carr,  C.,  & Anderson,  A.  (2001).  Computer-supported  collaborative  argumentation:  Supporting  problem-based 
learning  in  legal  education.  Paper  presented  at  the  Computer  Support  for  Collaborative  Learning 
(CSCL)  2001  Conference.  Retrieved  October  30,  2003:  http://www.mmi.unimaas.nl/euro- 
cscl/Papers/25  .pdf. 

Cho,  K.,  & Jonassen,  D.  (2002).  The  effects  of  argumentation  scaffolds  on  argumentation  and  problem  solving. 
Educational  Technology  Research  and  Development,  50  (3),  5-22.  Retrieved  from 
http://tiger.coe.missouri.edu/~jonassen/Argumentation.pdfon  March  3,  2004. 

Duffy,  T.  M.,  Dueber,  B.,  & Hawley,  C.  L.  (1998).  Critical  thinking  in  a distributed  environment:  A 

pedagogical  base  for  the  design  of  conferencing  systems.  In  C.  J.  Bonk,  & K.  S.  King  {Eds.),  Electronic 
collaborators:  Learner-centered  technologies  for  literacy,  apprenticeship,  and  discourse  (pp.  51-78). 
Mahwah,  NJ:  Erlbaum. 

Eleuterio,  M.,  Barthes,  J.  & Bartolozzi,  F.  (2002).  Mediating  collective  discussions  using  an  intelligent 

argumentation-based  framework.  In  Proceedings  of  the  Computer  Support  for  Collaborative  Learning 
(CSCL)  2002  Conference. 

Fahy,  P.  (2002a).  Epistolary  and  expository  interaction  patterns  in  a computer  conference  transcript.  Journal  of 
Distance  Education,  17(1),  20-35. 

Fahy,  P.  (2002b).  Use  of  linguistic  qualifiers  and  intensifiers  in  computer  conference.  The  American  Journal  of 
Distance  Education,  7(5(1),  5-22. 

Fahy,  P.  (2003).  Indicators  of  support  in  online  interaction.  International  Review  of  Research  in  Open  and 

Distance  Learning , 4(1).  Retrieved  April  21,  2004  from:  http://www.irrodl.Org/content/v4.l/fahy.html. 
Garrison,  R.  (2000).  Theoretical  challenges  for  distance  education  in  the  21st  century:  A shift  from  structural  to 
transactional  issues.  International  Review  of  Research  in  Open  and  Distance  Learning,  1 (1),  1-17. 


401 


Gottman,  J.  M.  (1979).  Marital  interaction:  Experimental  investigations.  New  York,  Academic  Press. 

Gunawardena,  C.,  Lowe,  C.,  & Anderson,  T.  (1997).  Analysis  of  global  on-line  debate  and  the  development  of 
an  interaction  analysis  model  for  examining  social  construction  of  knowledge  in  computer 
conferencing.  Journal  of  Educational  Computing  Research,  17,  397-431. 

Herring,  S.  (1993).  Gender  and  democracy  in  computer- mediated  communication.  Electronic  Journal  of 

Communication,  3(2).  Retrieved  August  22,  2001  from:  http://www.cios.org/www/ejc/v3n293.htm 

Herring,  S.  (1996).  Two  variants  of  an  electronic  message  schema.  In  S.  Herring  (Ed.),  Computer-mediated 
communication:  Linguistic,  social  and  cross-cultural  perspectives,  (pp.  81-106).  Amsterdam:  John 
Benjamins 

Jacques,  P.,  Oliveira,  F.  & Vicari,  R.  (2002a).  An  experiment  using  software  agents  for  Dialogue  analysis  in 
collaborative  distance  learning.  In  Proceedings  of  the  Computer  Support  for  Collaborative  Learning 
(CSCL)  2002  Conference. 

Jacques,  P.,  Andrade,  A.,  Jung,  J.,  Bordini,  R.  & Vicari,  R.  (2002b).  Using  pedagogical  agents  to  support 

collaborative  distance  learning.  In  Proceedings  of  the  Computer  Support  for  Collaborative  Learning 
(CSCL)  2002  Conference. 

Jeong,  A.  (2003).  The  sequential  analysis  of  group  interaction  and  critical  thinking  in  on-line  threaded 
discussions.  The  American  Journal  of  Distance  Education,  77(1),  25-43. 

Jeong,  A.  (2004a).  Methods  and  Tools  for  the  Computational  Analysis  of  Group  Interaction  and  Argumentation 
in  Asynchronous  Online  Group  Discussions . Paper  submitted  to  the  Annual  meeting  of  the  American 
Educational  Research  Association,  Montreal,  Canada,  April  2005. 

Jeong,  A.  & Juong,  S.  (2004b).  The  effects  of  response  constraints  and  message  labels  on  group  interaction  and 
argumentation  in  online  discussions.  Paper  presented  at  the  American  Educational  Research 
Association  conference,  San  Diego  CA. 

Jonassen,  D.,  & Remidez,  H.  (2002).  Mapping  alternative  discourse  structures  onto  computer  conference.  Paper 
presented  at  the  Paper  presented  at  Computer  Support  for  Collaborative  Learning  2002  Conference: 
Foundations  for  a CSCL  Community,  Boulder,  CO. 

Karacapilidis,  N.  & Papadias,  D.  (2001).  Computer  supported  argumentation  and  collaborative  decision 

making:  The  Hermes  system.  In  Proceedings  of  the  Computer  Support  for  Collaborative  Learning 
( CSCL)  2001  Conference. 

Levin,  J.,  Kim,  H.  & Riel,  M.  (1990).  Analyzing  instructional  interactions  on  electronic  message  networks. 
Online  Education,  L.  Harasim  (ed.),  Praeger,  New  York,  pp.  185-213. 

McAlister,  S.  (2003).  Assessing  good  argumentation.  Retrieved  April,  10,  2004,  from  the  World  Wide  Web: 
http://iet.open.ac.Uk/pp/s.r.mcalister/personal/AssessingGEA.htm 

Newman,  D.,  Johnson,  C.,  Cochrane,  C.,  & Webb,  B.  (1996).  An  experiment  in  group  learning  technology: 

Evaluating  critical  thinking  in  face-to-face  and  computer  supported  seminars.  Interpersonal  Computing 
and  Technology:  An  Electronic  Journal  for  the  21st  Century,  4(1),  57-74. 

Olson,  G.,  Herbsleb,  J.,  & Rueter,  H.  (1994).  Characterizing  the  sequential  structure  of  interactive  behaviors 
through  statistical  and  grammatical  techniques.  Human-Computer  Interaction,  9(3/4),  427-472. 

Rourke,  L.,  Anderson,  T.,  Garrison,  D.  R.,  & Archer,  W.  (2001).  Methodological  issues  in  the  content  analysis 
of  computer  conference  transcripts.  International  Journal  of  Artificial  Intelligence  in  Education,  12,  8- 
22. 

Savicki,  V.,  Kelley,  M.,  & Ammon,  B.  (2002).  Effects  of  training  on  computer- mediated  communication  in 
single  or  mixed  gender  small  task  groups.  Computers  in  Human  Behavior,  18,  257-269. 

Sloffer,  S.,  Dueber,  B.,  & Duffy,  T.  (1999).  Using  asynchronous  conferencing  to  promote  critical  thinking:  Two 
implementations  in  higher  education.  Retrieved  on  October  30,  2003  at 
http://crlt.indiana.edu/publications/crlt99-8.pdf 

Smith,  C.,  McLaughlin,  M.  & Osborne,  K.  (1997).  Conduct  controls  on  Usenet.  Journal  of  Computer-Mediated 
Communication,  2(4).  Retrieved  March  3,  2004  from 
http://www.ascusc.org/jcmc/vol2/issue4/smith.html 

Stern,  D.  N.  (1974).  Mother  and  infant  at  play:  The  dyadic  interaction  involving  facial,  vocal,  and  gaze 

behaviors.  In  M.  Lewis  & L.  A.  Rosenblum  (Eds.),  The  effect  of  the  infant  on  its  caregiver  (pp.  187- 
213).  New  York:  Wiley. 

Toulmin,  S.E.  (1958).  The  uses  of  argument.  Cambridge:  University  Press. 

Veerman,  A.,  Andriessen,  J & Kanselaar,  G.  (1999).  Collaborative  learning  through  computer- mediated 

argumentation.  In  C.  M.  Hoadley  and  J.  Roschelle  (Eds.),  Proceedings  of  the  Computer  Support  for 
Collaborative  Learning  (CSCL)  1999  Conference  (pp.  640-650).  Palo  Alto,  CA:  Stanford  University 


402 


[Available  from  Lawrence  Erlbaum  Associates,  Mahwah,  NJ]. 


403 


Rural  Creativity:  A Study  of  District  Mandated  Online  Professional 

Development 

Cynthia  Johnson 

Jefferson  County  North  Unified  School  District  139 

Jennifer  Summerville 
University  of  North  Carolina 

Introduction 

According  to  the  annual  industry  report  in  “Training”  magazine,  money  spent  on  employee  training 
dropped  approximately  six  percent — the  first  time  that  training  expenditures  have  dropped  since  the  mid  1990’s. 
At  the  same  time,  web-based  training  increased  from  48%  of  all  computer-based  training  to  61%  in  just  one 
year  (2002-2003).  The  most  “bang  for  the  buck”  in  the  business  sector  is  in  one  area  of  rapid  growth — e- 
leaming.  (Galvin,  2003). 

School  districts  are  not  exempt  from  this  growing  trend.  Rodes,  Knapczyk,  Chapman  & Haejin  (2000) 
suggest  that  the  population  most  in  need  of  e-learning  is  “[cjontinuing  education  students,  teachers  in  rural 
areas,  and  inservice  personnel  in  need  of  professional  development”.  However,  how  well  “e-training”  meets  the 
needs  of  professional  educators  remains  to  be  seen. 

In  the  spring  of  2002,  the  professional  development  committee  and  administrative  team  of  a rural 
school  district  in  the  Midwest  decided  to  implement  an  online  staff  development  learning  opportunity.  Each 
educator  at  the  middle  and  secondary  education  level  would  enroll  in  and  complete  one  online  course  in  his  or 
her  chosen  discipline.  This  course  would  be  paid  for  by  the  school  district  as  part  of  staff  development. 

Background 

In  a forward-thinking  technology  district,  administration  is  constantly  challenged  to  come  up  with  new 
and  creative  ideas  for  faculty  and  staff  support  and  development.  During  a professional  development  committee 
meeting,  the  superintendent  offered  the  idea  of  online  staff  development.  Members  of  this  committee  included 
the  curriculum  director,  high  school  principal,  and  faculty  representatives.  The  faculty  representatives  included 
the  chairperson,  one  elementary  teacher,  one  middle-level  teacher,  one  high  school  teacher,  and  one  at-large 
teacher. 

The  committee  believed  the  online  opportunity  would  allow  teachers  to  become  more  familiar  with  the 
technology  they  had  available,  help  teachers  better  understand  the  experience  of  students  enrolled  in  online 
courses,  and  provide  specialized  training  that  would  not  otherwise  be  available.  The  committee  agreed  that  the 
online  staff  development  would  be  required  of  all  secondary  teachers. 

Although  all  the  teachers  had  laptops  and  technology  provided  by  the  district,  the  diversity  of  the 
faculty  also  included  a wide  range  of  integration  and  comfort -levels  issues  dealing  with  technology.  The 
flexibility  of  studying  a variety  of  subjects  online  might  improve  the  effectiveness  of  the  staff  development  as  it 
was  important  subject  matter  to  the  educators  taking  the  courses. 

Requiring  educators  who  were  unfamiliar  with  the  expectations  and  happenings  of  online  learning  to 
take  an  online  course  might  foster  negative  feelings.  Creating  a survey  to  analyze  how  the  faculty  felt  about  the 
online  experience  would  provide  helpful  information  for  future  district  online  staff  development  and  for  other 
districts  considering  the  implementation  of  online  professional  training. 

Methodology 

Subjects 

Subjects  were  not  chosen  at  random  as  this  was  a district  mandated  program.  Each  educator  teaching 
grades  six  through  twelve  was  required  to  select  and  complete  an  online  course;  kindergarten  through  fifth 
grade  educators  would  not  be  required  to  fulfill  this  requirement.  In  all,  23  secondary  educators,  two 
administrators  and  one  full-time  tutor  participated  in  this  program. 
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Technology  Training 

The  school  district  would  provide  the  technology,  time,  and  money  for  the  endeavor  into  online 
learning.  During  the  year  and  a half  prior  to  participating  in  the  online  program,  each  educator  was  given  a 
personal  laptop  and  received  training  on  how  to  use  the  laptop  properly. 

District-level  staff  development  time  scheduled  in  mid-August  before  school  began  was  utilized  to 
teach  the  faculty  how  to  set  up  and  use  the  computers.  One  session  was  held  for  the  elementary  and  middle  - 
level  faculty  while  another  session  was  held  for  the  high  school  faculty.  Additional  required  sessions  included 
learning  how  to  use  the  email  system  and  grading  programs.  As  the  programs  and  computers  were  updated, 
required  building-level  training  was  provided  during  scheduled  in-service  days. 

Online  Courses 

Although  educators  were  allowed  to  choose  their  courses  based  on  personal  preference,  most  educators 
chose  topics  that  were  specialized  according  to  his  or  her  particular  area  of  teaching.  For  example,  music 
educators  completed  “Internet  for  Music  Educators.”  The  remaining  educators  chose  broader  subjects  that 
would  benefit  the  school  as  a whole  or  were  more  traditional  in-service  topics.  These  included  such  topics  as 
classroom  management,  sexual  harassment,  and  curriculum  compacting. 

Of  the  26  participants,  22  educators  chose  online  courses  provided  by  a recommended  provider,  three 
educators  chose  graduate  courses  offered  by  a regional  university  and  one  educator  chose  an  alternative  online 
course  provided  by  an  additional  provider. 

The  recommended  courses  ranged  in  cost  from  $72  - $120;  20  of  the  22  courses  cost  $72.  The  three 
educators  who  chose  graduate  courses  each  enrolled  in  three-hour  courses  and  were  reimbursed  $180  total. 
University  graduate  courses  were  reimbursed  at  the  rate  of  $60  per  credit  hour  as  outlined  in  the  district’s 
negotiated  agreement.  The  approved  alternative  course  by  an  outside  provider  cost  $100,  which  was  paid  by  the 
district.  Courses  offered  through  the  recommended  provider  were  completely  paid  for  by  the  district. 

Courses  were  to  be  completed  by  May  with  the  exception  of  three  educators  enrolled  in  graduate-level, 
semester-length  courses.  The  remaining  educators  enrolled  in  courses  that  were  independent  in  nature  and 
allowed  the  educators  to  work  at  their  own  pace.  While  a few  were  able  to  complete  the  course  in  one  intensive 
day  session,  most  of  the  educators  completed  their  courses  in  two  months. 

Instrumentation 

The  data  collection  instrument  was  a survey  consisting  of  26  likert -scale  questions  and  four  open- 
ended/short  answer  questions.  Question  categories  included  course  effectiveness,  instructor  effectiveness,  self- 
perceptions of  the  educator  as  a student,  and  support  services  and  technology.  See  Appendix  A for  a copy  of 
the  full  survey.  A source  of  reference  was  an  online  course  survey  from  a regional  university  that  was 
completed  by  students  near  the  end  of  a semester  online  course.  Particular  points  of  interest  from  the 
professional  development  standpoint  included  whether  the  faculty  felt  comfortable  with  the  technology,  whether 
the  online  format  was  effective,  and  would  the  faculty  consider  enrolling  in  another  online  course. 

Data  Collection 

The  survey  was  sent  via  an  email  attachment  and  hard  copies  were  also  placed  in  each  educator’s 
mailbox.  88  percent  (23/26)  of  the  participants  completed  and  returned  the  online  staff  development  survey.  21 
respondents  returned  the  hard  copy  that  was  placed  in  their  mailboxes,  one  respondent  printed  off  the 
attachment  and  sent  in  the  completed  survey,  and  one  respondent  returned  the  completed  email  attachment. 

Results 

As  displayed  in  Figure  1,  the  majority  of  the  questions  in  the  first  section  titled  Course  Effectiveness 
received  high  marks.  Respondents  believed  the  syllabus  accurately  described  course  content  and  objectives  (Ql) 
and  that  the  course  pace  and  difficulty  was  appropriate  (Q3)  and  assignments  were  reasonable  and  appropriate 
(Q2). 
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Figure  1 

Course  Fffectiveness 


Interaction  with  fellow  students  (Q5)  and  interaction  with  the  instructor  (Q6)  received  relatively  low 
rankings.  Two  respondents  noted  that  there  was  no  interaction  between  fellow  students  and  no  interaction  with 
the  instructor.  Respondents  agreed,  as  indicated  by  the  4.00  average  that  the  online  course  increased  their 
interest  in  the  subject  (Q8)  and  they  would  recommend  the  course  to  other  students  (Q7). 

In  the  second  section.  Instructor  Effectiveness,  overall  effectiveness  as  a teacher  and  facilitator  of 
online  learning  (Q 11)  received  mostly  positive  responses  as  did  the  class  was  well-prepared  with  stimulating 
lessons  (Q13)  and  used  grading  procedures  that  were  fair  and  equitable  (Q 17). 


Questions  16,  18,  and  19  all  received  2.88  and  below  ratings.  Respondents  did  not  feel  like  instructors 
commented  on  their  work  in  ways  that  helped  them  learn,  instructors  did  not  realize  when  students  did  not 
understand,  and  instructors  were  not  willing  to  help  students  outside  of  the  class.  Five  respondents  did  not 
answer  question  16  because  instructors  did  not  comment  at  all  on  assignments  or  coursework.  Six  respondents 
did  not  answer  questions  17  - 19  concerning  fair  grading  practices,  realizing  when  students  did  not  understand, 
and  helping  outside  of  class.  Two  respondents  commented  there  was  not  an  instructor  presence  because  it  was 
an  online  course. 

In  the  section  on  Student  Preparedness,  graphed  in  Figure  3,  all  of  the  questions  received  relatively 
positive  ratings.  Question  20  rated  interest  in  taking  the  course,  which  the  respondents  rated  as  a 4.00. 

The  next  question  related  to  effort  applied  toward  learning,  which  remained  high  even  though  the 
course  was  required.  The  amount  learned  in  the  course  received  the  lowest  rating  of  the  student  section  with  a 
3.70.  One  respondent  commented  that  the  course  was  better  suited  for  a new  student  instead  of  an  experienced 
educator.  Confidence  in  using  the  learned  information  was  the  highest  rating  in  the  section  with  a 4.04.  The 
freedom  to  choose  any  course  allowed  educators  to  select  online  courses  that  were  relevant  for  their  particular 
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situations  or  teaching  fields. 


Question  23  rated  prior  computer  experience,  which  received  a rating  of  3.87.  Each  educator  was 
provided  a laptop  in  the  fall  of  2000;  educators  had  the  computers  for  one-and-a-half  years  before  the  online 
requirement.  Subsequent  in-service  opportunities  have  been  provided  and  technology  support  staff  was 
available  for  additional  help  during  the  regular  workday.  The  last  question  in  the  student  section  asked  if  the 
respondent  would  consider  enrolling  in  another  online  course.  With  a score  of  3.87,  respondents  leaned  toward 
agreement. 

The  ability  to  receive  technical  assistance  from  appropriate  support  services  received  a rating  of  3.52. 
Respondents  were  able  to  use  support  services  from  the  online  provider,  the  professional  development 
committee,  and  the  technology  support  staff.  Respondents  also  referred  to  other  educators  for  assistance. 

Results  and  Relationship  to  Research 

Successes  of  Online  Experience 

Flexibility  of  Time  Five  respondents  specifically  mentioned  flexibility  of  completing  the  course  as  a benefit  of 
online  learning.  Educators  could  use  the  two  in-service  days  set  aside  for  the  course  or  they  could  use  their  own 
personal  time.  Coursework  could  be  completed  at  work  or  it  could  be  completed  while  at  home.  Since 
educators  each  had  a laptop,  they  could  work  on  their  online  course  anywhere  they  could  find  an  Internet 
connection.  The  courses  were  also  asynchronous  so  educators  could  log  on  at  any  time  of  the  day. 

Online  staff  development  also  provides  “just  in  time”  learning  when  educators  need  it  most 
(Richardson,  2001).  The  increased  accessibility  allows  educators  to  log  on  at  anytime  but  it  also  allows  them  to 
retain  online  resources  and  refer  to  them  at  anytime  during  the  semester  or  in  the  future  (Barkley  and  Bianco, 
2000). 

Self-Paced  Courses  Five  respondents  also  added  that  working  at  their  own  pace  was  a benefit  of  the  online 
learning  atmosphere.  Some  educators  completed  their  courses  in  one  session  on  one  of  the  provided  in-service 
days  while  others  worked  on  the  courses  as  it  was  convenient  to  their  schedules. 

The  self-paced  atmosphere  also  allowed  educators  the  privacy  of  reviewing  materials  they  did  not  feel 
they  mastered.  Educators  did  not  have  to  worry  about  fellow  peers  realizing  they  were  reviewing  past 
information.  “Constructivist  research  has  demonstrated  that  teachers,  as  well  as  students,  generally  prefer  to  be 
in  charge  of  their  own  learning  and  prefer  to  build  their  own  knowledge,”  added  Odasz  in  his  article  on  Alaskan 
professional  development  (1999). 

Educators  enjoy  the  self  pace  because  they  can  move  as  quickly  or  as  slowly  as  they  feel  is  necessary. 
Odasz  also  added  that  once  the  teachers  were  comfortable  with  the  instructions  and  course,  they  appreciated 
“not  being  specifically  led,  but  being  left  to  learn  on  their  own,  in  their  own  way”  (1999). 

Viable  Content  and  Information  The  district  allowed  the  educators  to  choose  what  course  they  would  enroll  in 
for  their  online  learning  experience.  By  providing  the  educators  with  the  topic  choice,  educators  could  choose 
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whatever  topic  they  felt  would  benefit  them  the  most.  The  educator  was  also  allowed  to  select  the  provider, 
which  allowed  more  opportunity  for  specialization  and  refinement  of  content. 

One  technology  coordinator  agreed  that  “tailored  training  would  certainly  be  more  motivation  to  our 
staff  than  the  en  masse  training  that  is  currently  typical”  (Zahner,  2002).  Requiring  the  online  staff 
development  and  allowing  individual  educators  to  select  course  topics  provided  a wonderful  opportunity  for 
customized  learning  for  each  educator’s  needs  (Richardson,  2001). 

New  Ideas  In  addition  to  useful  content,  the  online  courses  also  provided  the  educators  with  the  opportunity  to 
meet  new  colleagues  and  share  new  ideas.  Online  collaborative  tools  allowed  the  educators  to  “develop  new 
insights  into  pedagogy  and  their  own  practice”  and  to  “explore  new  or  advanced  understandings  of  content  and 
resources”  (Zahner,  2002).  Educators  were  able  to  engage  in  learning  with  other  online  students  with  similar 
interests  and  to  “facilitate  the  exchange  of  ideas  and  information”  (Simkins,  1998). 

Considerations  for  Improvement 

Online  Course  Offerings  Some  educators  commented  they  did  not  have  enough  information  on  how  to  find  an 
online  course  or  evaluate  the  providers  of  the  online  courses.  One  respondent  specifically  mentioned  that  a list 
of  available  courses  and  a synopsis  of  content  would  have  been  helpful  during  the  selection  process.  Although 
there  was  an  information  packet  distributed  to  the  participants,  little  information  was  included  on  the  content  of 
specific  courses. 

The  majority  of  the  educators  enrolled  in  the  recommended  courses  primarily  because  they  were 
recommended  in  the  notification  and  the  tuition  would  be  completely  covered  by  the  school  district.  One 
educator  did  research  an  additional  online  course  provider  but  encountered  problems  logging  in,  accessing  the 
course,  and  receiving  assistance. 

A list  of  colleges  with  graduate  level  online  courses  would  also  be  helpful  as  well  as  web  addresses  to 
online  course  offerings.  Web  sites  that  provide  links  to  example  course  pages  would  also  help  educatois  make  a 
more  informed  choice  on  the  type  of  online  course  and  level  of  interaction.  Educators  could  also  use  these 
graduate  hours  to  recertify  with  the  state  department. 

Support  Staff  Technology  support  staff  was  available  during  the  course  of  the  semester  but  the  hours  of 
availability  were  somewhat  limited.  Each  member  of  the  tech  support  staff  was  also  a part-time  teacher  and  had 
teaching  responsibilities  throughout  the  regular  workday.  Educators  who  did  not  share  plan  time  or  tech 
support  time  with  a support  staff  member  had  difficulty  receiving  help  during  the  workday. 

Time  was  available  if  educators  were  willing  to  report  before  or  after  school.  Odasz  commented, 
“Anxiety  is  reduced  knowing  help  is  readily  available,  anytime  they  need  it”  (1999).  If  a support  staff  member 
was  available  during  each  hour  of  the  workday,  this  would  decrease  anxiety  and  help  solve  problems  while  they 
were  still  small  and  manageable.  Educators  would  also  be  able  to  seek  help  and  advice  during  the  regular 
workday  and  would  not  have  to  make  appointments  before  or  after  school. 

Also  an  initial  meeting  with  all  participants  could  have  eased  the  transition  into  online  learning.  Many 
of  the  problems  encountered  by  the  educators  were  problems  with  the  initial  log  in  and  compatibility  issues  with 
browsers.  An  initial  meeting  with  tech  support  could  have  provided  an  opportunity  for  educators  to  log  in,  try 
passwords,  and  choose  the  best  browser  for  the  online  course. 

Time  Allowance  Most  educators  who  did  not  enroll  in  graduate-level,  semester  courses  spent  an  average  of  10.8 
hours  on  the  course.  Two  other  educators  estimated  it  took  20-30  hours  to  complete  the  online  coursework. 

The  majority  of  the  participants  worked  on  the  course  during  their  personal  time.  Working  on  the  course  during 
personal  time  may  have  also  made  it  feel  like  there  was  not  enough  compensated  time  (a  total  of  two  inservice 
days)  set  aside  to  complete  the  assignments  and  the  necessary  research. 

Facilitator  and  Student  Interaction  The  majority  of  the  respondents  felt  their  online  experience  could  have 
been  improved  with  more  interaction  from  the  facilitator  and  fellow  students.  23  of  the  online  courses  chosen 
by  the  professional  development  committee  were  designed  as  self-paced,  independent  lessons  for  individual 
professional  development.  The  benefit  was  the  educator  could  work  through  the  content  at  his  or  her  own  pace. 

The  disadvantage  was  the  lack  of  interaction.  Educators  did  not  feel  like  there  was  a facilitator  or 
instructor  for  the  course,  and  there  was  very  little  interaction  between  the  educators  enrolled  in  the  courses. 

One  high  school  educator  commented,  “If  I want  to  read  more  about  it,  I’ll  go  to  the  library  and  check  out  a 
book  and  read  it  at  home,  on  my  deck  or  in  my  recliner,  rather  than  at  a keyboard.” 


408 


Educators  enrolled  in  graduate  courses  were  able  to  experience  more  interaction  between  the  facilitator 
and  the  fellow  students  enrolled  in  the  course.  The  graduate  courses  used  more  interactive  collaborative  tools 
such  as  discussion  forums  and  chat  rooms.  The  graduate  courses  also  included  assignments  that  required  group 
projects  in  which  the  educators  had  to  work  online  with  fellow  educators  to  complete  assignments. 

Student -to-student  interaction  plays  a vital  role  in  online  learning.  Students  can  post  questions  to 
discussion  forums  and  have  peers  offer  assistance.  This  student-to-student  interaction  improves  the 
communication  between  the  students  during  the  course,  helps  relieve  the  number  of  questions  the  facilitator 
must  answer,  and  builds  colleague  relationships  that  will  extend  beyond  the  length  of  the  online  course. 

As  online  students,  educators  learn  how  to  use  online  collaborative  tools  to  exchange  ideas  with  peers; 
continuing  to  use  these  communication  tools  will  allow  educators  to  stay  more  up-to-date  with  the  ever- 
changing  world  of  education.  This  new  opportunity  for  peer  collaboration  and  simply  “talking”  with  fellow 
teachers  is  something  teachers  often  cannot  find  time  for  during  the  standard  school  day  ( Killion,  2000). 

Conclusions  and  Suggestions  for  Future  Research 

Overall,  requiring  educators  to  take  an  online  course  as  part  of  staff  development  appears  to  have  been 
a positive  endeavor.  In  an  environment  where  time  is  often  an  opponent  and  funding  for  staff  development  is 
scarce,  it  is  important  to  find  alternatives  for  providing  training  opportunities. 

However,  several  issues  arose  during  the  analysis  of  the  data  for  this  study.  These  issues  may  be 
important  for  future  research.  First,  what  types  of  information  should  be  provided  so  that  educators  could  make 
informed  decisions  about  online  courses?  How  can  this  information  be  provided  in  an  unbiased  manner  without 
appearing  to  advocate  certain  courses? 

Next,  should  a "starter”  online  course  be  provided  before  taking  the  required  course?  In  other  words, 
would  a "how  to  learn  online"  course  be  of  benefit  to  the  overall  feelings  of  success  in  online  course 
environments?  If  so,  how  would  this  be  developed,  delivered  and  funded? 

Finally,  how  much  time  should  be  allotted  for  staff  development?  Should  additional  time  be  allotted 
and  funded  (as  this  was  a district  requirement)  or  is  the  traditional  allotted  time  (one  or  two  inservice  days) 
sufficient  for  online  courses?  Should  the  district  provide  alternatives  other  than  inservice  days  (such  as  a 
monetary  incentive)? 

It  is  the  opinion  of  the  authors  of  this  article  that  online  venues  can  provide  ideal  training  alternatives 
for  educators  in  rural  or  other  districts  who  may  not  have  access  to  the  varied  courses  available  in  larger 
metropolitan  areas.  Districts  considering  such  proposals  will  need  to  be  aware  of  potential  issues  over  which 
they  have  control — time,  compensation  and  information.  Armed  with  this  knowledge,  we  hope  educators  and 
administrators  can  make  informed  decisions  about  implementing  online  staff  training  and  development. 
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Online  Staff  Development  Evaluation 


Appendix  B 


Name: Position: 

Course  Title: Course  Provider: 

How  long  did  it  take  you  to  complete  the  course? 

Please  give  honest  and  thoughtful  answers  to  the  following  questions.  The  evaluation  contains  five  sections. 
The  first  section  is  course  effectiveness.  Section  two  is  instructor  effectiveness.  The  third  section  relates  to  you 
as  a student.  Support  services  and  technology  is  the  fourth  section.  The  final  section  contains  open-ended 
questions.  Please  respond  to  the  questions  in  each  section.  You  may  omit  an  item  if  you  feel  unable  to  make  a 
fair  judgement. 

Please  rate  your  level  of  agreement  using  the  following  scale: 

1 2 3 4 5 

Strongly  Disagree  Disagree  No  Opinion/Neutral  Agree  Strongly  Agree 


COURSE  EFFECTIVENESS 

4 Syllabus  accurately  described  course  content  and  objectives  1 

5 Assignments  were  reasonable  and  appropriate  1 

6 Course  pace  and  difficulty  were  appropriate  1 

7 Exams  and  quizzes  reflected  important  course  aspects  1 

8 Level  of  interaction  with  fellow  students  1 

9 Level  of  interaction  with  the  instructor  1 

10  I would  recommend  this  course  to  other  students  1 

11  The  course  increased  my  interest  in  the  subject  1 

12  Having  completed  the  course,  I feel  knowledgeable 

in  the  subject  1 

13  Overall,  the  course  and  instructor  met  my  expectations  1 


2 

2 

2 

2 

2 

2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 

5 

5 

5 

5 


INSTRUCTOR  EFFECTIVENESS 


14  Overall  effectiveness  as  a teacher  and  facilitator  of 

online  learning  1 

15  Making  clear  the  goals  and  objectives  of  this  online  course  1 

16  Being  well  prepared  for  the  class  (ex.  designing  well 

planned  lessons  and  activities)  1 

17  Explaining  the  subject  matter  so  you  understand  1 

18  Stimulating  you  to  think  more  deeply  about  the  subject 


2 

2 

2 

2 


3 

3 

3 

3 


4 

4 

4 

4 


5 

5 

5 

5 
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(applying  information,  analyzing,  solving  problems) 

19  Commenting  on  your  work  (tests  and  assignments)  in 
ways  that  helped  you  learn  (ex.  online  discussions,  etc). 

20  Using  grading  procedures  that  were  fair  and  equitable 

21  Realizing  when  students  did  not  understand 

22  Being  willing  to  help  students  outside  of  class  (give 
assistance  via  email,  phone,  virtual  office  hours,  and 
supplemental  mailings) 


1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 


YOU  AS  A STUDENT 


23 

Your  interest  in  taking  this  course  before  you  enrolled 

1 

2 

3 

4 

5 

24 

Your  effort  to  learn  in  this  course  (studying,  completing 
assignments,  brainstorming  ideas) 

1 

2 

3 

4 

5 

25 

The  amount  you  have  learned  in  this  course 

1 

2 

3 

4 

5 

26 

Your  computer  experience  prior  to  this  course 

1 

2 

3 

4 

5 

27 

How  confident  do  you  feel  about  using  the  information 
presented  in  this  course 

1 

2 

3 

4 

5 

28 

Would  you  enroll  in  another  online  course? 

1 

2 

3 

4 

5 

SUPPORT  SERVICES  & TECHNOLOGY 

29  Your  ability  to  receive  technical  assistance  from  the 

appropriate  support  services  1 2 3 4 5 

30  What  could  be  done  to  make  it  easier  for  you  to  be  a distance  learner? 


31  Describe  any  frustrations  or  problems  with  technology  in  this  course. 


32  What  did  you  like  best  about  this  online  course? 


33  What  did  you  like  least  about  this  online  course? 


Thank  you 
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Joan  Mazur 

University  of  Kentucky 

[Note:  the  appendices  cited  in  this  document  are  not  included,  but  can  be  found  online  at 
http://www.freedomseas.net/Pro/Pro.htm] 

Introduction 

Computer  mediated  communication  has  become  a regular  part  of  most  teachers’  professional  daily 
routines.  In  a relatively  short  amount  of  time  (a  mere  heartbeat  when  compared  to  the  speed  with  which  many 
other  media  came  into  regular  use),  email  has  emerged  as  a primary  method  of  communication  by  which 
administrators  disseminate  important  information  to  a school’s  teachers,  appointments  are  made,  notices  posted, 
and  professional  connections  are  established  and  maintained  between  educational  professionals  - within  a 
school  or  the  rest  of  the  outside  world.  Not  surprisingly,  teachers  involved  in  online  instruction  have  embraced 
computer-mediated  communication  as  the  method  of  choice  for  providing  class  interaction  and  opportunities  for 
knowledge  construction  within  the  course  context. 

Online  forums  that  purport  to  offer  access  to  learning  communities  for  practicing  teachers  have 
proliferated  in  recent  years.  The  net-seminars  offered  through  the  Concord  Consortium  (Tinker  & Haavind, 
1996),  Tapped  ln(Schlager  & Schank,  1997),  the  Maryland  Electronic  Learning  Community  (Rose,  Allen,  & 
Fulton,  1999)  and  Teachers. Net  (Kovaric  & Bott,  2000)  are  among  the  first  groups  to  focus  on  providing 
practicing  teachers  with  opportunities  for  professional  conversations  about  teaching  and  learning.  For  some  time 
though,  researchers  have  noted  disparities  between  the  promise  of  community  and  professional  dialogue  for 
participants  in  these  online  forums  and  the  actual  practice  and  experience  (Harrington,  Quinn-Leering,  & 
Hodson,  1996)  of  those  who  join  such  groups.  Issues  regarding  the  quality  of  both  the  content  and  level  of 
critical  discourse,  as  well  as  the  authenticity  of  the  ‘community’  that  is  actually  established,  have  been  raised 
(Whipp,  2004).  A closer  examination  is  needed  regarding  how,  or  if,  the  nature  of  the  interaction,  and  the 
content  of  conversation,  can  result  in  viable  online  communities  that  support  professional  practice. 

For  all  teachers,  an  extensive  network  of  communication  existing  for  the  sole  purpose  of  social 
interaction  and  entertainment  has  also  arisen,  and  exists  sometimes  superimposed  on  top  of,  or,  perhaps  more 
accurately,  just  beneath,  their  professional  discourse.  Within  many  of  these  contexts,  the  potential  for  a melding 
of  social/personal  interaction  and  professional  discussion  into  an  active  attempt  at  community  building  through 
electronic  discourse  seems  a very  real  possibility,  a natural  outgrowth  of  the  capabilities  of  this  online 
communication.  Unfortunately,  there  are  few  stories  documenting  the  successful  use  of  such  tools  in  creating, 
structuring,  even  encouraging  teacher  communities  with  a shared  sense  of  identity  and  purpose.  Since  electronic 
discourse  brings  with  it  the  advantage  of  instantly  archivable  sources  of  data,  the  motivation  to  try,  document, 
and  analyze  such  community-building  activities  (thereby  contributing  to  the  closing  of  this  research  gap)  has 
never  been  higher.  In  this  paper  we  will  examine  one  such  activity.  In  the  process,  we  hope  to  contribute  to  an 
expansion  of  the  analytical  approaches  available  to  assess  the  success  by  which  this  type  of  activity  contributes 
to,  or  reflects,  a commu  nity  of  practice  amongst  teachers  with  a shared  purpose  across  a school  district. 


The  Need  for  Teacher  Community,  and  the  Context  of  Computer  Mediation 

A sense  of  professional  community  amongst  teachers  can  be  a pivotal  part  of  the  educational 
landscape.  Teacher  communities  can  reinforce  negative  stereotypes  and  poor  habits,  but  more  likely  they  will 
allow  a joint  development  of  teacher  practice,  and  a sharing  of  resources  and  traditions  (Barab,  MaKinster, 
Moore,  & Cunningham,  2001;  Beck  & Kosnik,  2001;  Grossman,  Wineburg,  & Woolworth,  2001;  Hara,  2001; 
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McLaughlin  & Talbert,  2001).  At  the  high  school  level,  existing  communities  of  practice  tend  to  coincide  with 
disciplines  or  departments,  and  “...how  traditions  of  teaching  play  out  in  the  classroom  depends  on  the  strength 
and  the  character  of  [these  communities]”  (McLaughlin  & Talbert,  2001,  64).  However,  it  is  entirely  too  easy  to 
assume  the  existence  of  professional  communities  of  practice  for  teachers,  where  there  may  be  little  that  is 
observable  of  what  we  normally  ascribe  to  the  phrase  (Grossman  et  al.,  2001).  Teachers  may  very  well  want  to 
connect  to  each  other,  but  such  connections  do  not  always  develop  naturally,  and  the  natural  groupings  present 
in  schools  - departments,  grade  levels,  shared  space,  even  the  school  itself  - may  not  successfully  create  a 
sufficient  sense  of  community  to  be  useful  to,  and  influential  on,  classroom  practice  (Carver  & Neuman,  1999). 

Of  course,  the  reasons  for  this  lack  of  connection  may  be  as  sinister  as  poor  preparation  for  the  process 
of  teacher  community  building  in  teacher  preservice  training,  or  as  pedestrian  as  a simple  lack  of  time  or  forums 
for  community-building  interaction.  It  is  in  this  context  that  we  look  to  computer  connectivity  and  text  -based 
computer  mediated  communication  to  step  forward.  Although  LISTSERV-style  email  discussion  forums  and 
other  communications  formats  nearly  span  the  history  of  computer  connectivity,  there  are  few  examples  of  how 
such  interaction  might  work  with  an  informal  connection  between  teachers  within  a district  who  share  a 
commonality  of  purpose  and  responsibility  across  school  boundaries.  Barab,  MaKinster,  et.al.  (2001)  included 
the  possibility  of  on-line  forum  interaction  into  their  electronic  Inquiry  Learning  Forum,  a web-delivered  entity 
which  deliberately  attempted  to  establish  a community  with  a plurality  of  purpose  - an  exploration  of  shared 
interests  as  well  as  an  exploration  of  the  concept  of  online  communication  and  community  itself.  However,  of 
the  three  initiatives  from  this  project  (“Visit  the  classroom  metaphor,”  “Knowledge  Creation-Management- 
Networking,”  and  “Commitment  to  Community”),  the  latter  - the  encouragement  of  a community  of  practice 
between  participants  - proved  to  be  the  most  challenging  of  the  goals  of  the  project.  Participation  in  the 
interactive  aspects  of  this  initiative  often  depended  on  structured  relationships,  such  as  class  requirements.  The 
designers  despaired  of  ever  reaching  the  “critical  mass”  required  to  achieve  the  goals  of  community  amongst 
veteran  teachers  (Barab  et  al.,  2001,  92). 

Threaded  online  discussions  are  a regular  feature  of  online  courses,  but  even  within  the  context  of 
structured  relationships  through  class  participation  and  required  interaction,  one  cannot  assume  that  such 
communication  will  lead  towards  connections  associated  with  community.  In  a graduate  course  for  inservice 
teachers,  Merryfield  (2001),  discovered  that  “...teachers  wrote  about  highly  valuing  the  online  technologies  for 
creating  a place  for  frank  discussion,  and  then,  even  within  the  same  message  or  on  the  next  day,  they  described 
those  same  technologies  as  barriers  that  kept  them  from  "knowing’  one  another  or  having  ‘real’  relationships” 

(p.  295).  Herring  (1999)  notes  such  a dichotomy  of  attitude  as  well  while  examining  synchronous  chat  and  other 
text -based  computer  mediated  communication.  Although  such  media  remain  incredibly  popular,  a cursory 
analysis  often  shows  little  useful  communication  and  meaningful  interaction  which  might  imply  community 
building.  One  might  assume  that  a designated  topical  purpose  in  an  asynchronous  environment  might  encourage 
more  useful  and  cohesive  discourse,  but,  as  Herring  further  remarks,  “[such  forum]  discussions  also  tend 
towards  topic  decay”  (Herring,  1999,  13). 

There  are,  of  course,  lots  of  examples  of  the  successful  use  of  computer  mediated  communication  in 
course  structures  (King,  2001;  Sujo  de  Montes  & Gonzales,  2000),  but  overall,  the  use  of  such  media  to  build 
community  within  a district  or  school  context  through  a free  discussion  of  issues  and  interests  is  extremely  rare, 
or  at  least  rarely  examined,  since  there  is  a substantial  lack  of  research. 

The  FCPS-TRT-L  Discussion  List 

The  focus  of  the  research  for  this  paper  is  the  email  discussion  list  FCPS  -TRT -L,  a LISTSERV-style 
forum  implemented  through  capability  provided  by  Fayette  County  Public  Schools  (Lexington,  Kentucky) 

Office  of  Technology.  The  forum  included  technology  resource  teachers  (TRT)  - both  those  hired  by  and  based 
in  schools,  and  those  hired  by  the  educational  support  arm  of  the  Fayette  County  Office  of  Technology  to  float 
between  schools.  School-based  Microcomputer  Resource  Technicians  (MRT)  - a position  found  in  many 
elementary  schools,  and  a few  middle  schools  - were  also  included,  even  though  MRTs  are  not  generally 
certified  teachers.  However,  because  of  their  primary  responsibility  as  lab  managers,  they  often  are  called  upon 
to  perform  instructional  support  duties,  making  their  interests  much  the  same  as  TRTs. 

The  issue  of  successful  integration  of  technology  into  instruction  is  important  to  Fayette  County,  as 
well  as  the  Kentucky  Department  of  Education  and  its  Educational  Professional  Standards  Board,  as  reflected  in 
the  latter’s  New  and  Experienced  Teacher  Standards  (EPSB,  1999).  In  addition,  Fayette  County  initiated  a set  of 
technology  curriculum  standards  for  students,  which  were  passed  by  the  School  Based  Decision  Making 
councils  of  all  district  schools  during  the  end  of  the  previous  decade.  The  Office  of  Educational  Technology  is 
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the  single  largest  district-supplied  resource  for  Fayette  County  schools  and  teachers,  further  illustrating  the 
commitment  of  the  district  towards  the  importance  of  technology  integration  into  the  classroom.  Connecting 
and  supporting  school-based  front-liners  such  as  TRTs  and  MRTs  was  viewed  as  an  important  function  of  the 
Office  of  Educational  Technology. 

The  FCPS-TRT-L  discussion  list  was  piloted  in  the  fall  of  2002  to  fill  this  educational  technology  integration 
support  niche.  Previously,  this  need  was  addressed  by  semi-monthly  face-to-face  meetings,  but  since 
participation  in  the  meetings  was  sparse  (many  had  only  one  or  two  in  attendance),  it  was  decided  to  try  another 
format,  and  a new  email  discussion  list  capability  put  on  line  a few  months  before  was  selected.  Unlike  most 
email  discussion  lists,  FCPS-TRT-L  is  a closed  forum  - that  is,  membership  was  pre-loaded  (all  TRTs  and 
MRTs  were  identified  and  included  in  the  membership  of  the  list  in  advance),  and  members  have  to  request  to 
be  added  or  leave  the  group.  As  the  list’s  creator,  I served  as  its  moderator,  and  initiated  many  of  the  threads  for 
discussion,  although  the  need  for  that  role  began  to  drop  off  as  the  list  began  to  develop. 

Conceptual  Framework  for  Research 

One  goal  of  the  email  discussion  list  was  to  attempt  to  establish  and  encourage  a community  of 
practice  between  educational  technology  integration  front-liners.  For  the  purpose  of  this  study,  “community  of 
practice”  will  follow  the  description  outlined  by  Wenger  (1998),  that  is,  we  would  expect  a successful 
community  to  exhibit: 

V mutual  engagement  - specifically,  the  ability  of  members  of  the  community  to  interact  in  such  a way 
that  it  displays  complementary  participation  and  provides  for  specialization,  as  an  extension  of  practice 
rather  than  ideas  or  structures, 

V through  the  reflective  process  of  negotiation,  evidence  of  an  indigenous  joint  enterprise  with  mutual 
accountability,  and 

V a shared  repertoire  of  actions  resulting  from  engagement. 

The  result  would  be  a community  which  can  present  and  solve  problems  relating  to  its  duties,  to 
discuss  policy  and  practice  in  the  classroom  and  lab,  to  introduce  new  capabilities  and  examine  resources,  in  a 
format  that  encourages  collegiality  and  a sense  of  shared  experience  and  purpose  (Hunt,  1999). 

It  was  also  the  intent  of  the  design  of  FCPS-TRT  -L  to  have  members  explore  the  email  discussion 
format  itself,  with  an  ultimate  goal  of  encouraging  its  use  directly  in  the  classroom.  In  addition  to  this 
previously -mentioned  purpose  of  the  Inquiry  Learning  Forum  (Barab  et  al.,  2001),  this  duality  of  purpose 
matches  very  closely  the  definition  of  community  illustrated  by  Grossman  and  Wineburg’s  work  with  Seattle 
English  and  history  teachers  (Grossman  et  al.,  2001).  In  that  work,  another  perceived  benefit  of  such  a building 
of  community  is  the  encouragement  of  instructional  leadership,  a further  goal  of  FCPS-TRT-L  and  (not 
incidentally)  a part  of  Kentucky  Teacher  Standards  (EPSB,  1999). 

Research  Design 

The  data  available  for  use  in  examining  this  on-line  community  were  nine  simple  text  files  generated 
by  the  list  management  software,  taken  during  the  weeks  between  the  12th  of  September  and  the  15th  of  October 
2002,  constituting  the  first  8+  weeks  of  discourse  for  the  list.  Because  of  budget  constraints,  a free  list 
management  package  was  selected  by  Fayette  County  to  support  email  discussion  lists  - David  Harris’s 
Mercury  Mail  Transport  System.  This  system  produces  raw  text  archives  which  are  not  in  a format  conducive  to 
analysis,  or  even  comfortable  reading.  Hence  a simple  Visual  Basic  routine  was  written  to  strip  out  extraneous 
header,  inclusion,  and  formatting  text,  and  to  isolate  and  define  important  data  fields  (including  date -time 
stamps  and  participants)  for  later  processing  (see  Appendix  D).  The  results  were  not  as  readably  available  and 
instantly  useable  as  the  on-line  searchable  list  databases  normally  a part  of  commercial  list  management 
packages  - text  files  had  to  be  downloaded  and  converted  locally  - but  the  results  were  exportable  to  an 
Access2000  database,  which  allowed  for  a variety  of  queries  and  report  formats  not  usually  available  on  line. 
Even  so,  some  direct  handling  of  the  data  was  still  required  - most  notably  because  of  configuration  decisions 
made  early  on  and  subsequently  corrected.  In  the  interests  of  a mixed  method  analytical  approach  (Tashakkori 
& Teddlie,  1998),  the  resultant  data  was  then  used  to  create  two  images  of  the  interactive  community  of  teacher 
participants  - a collection  of  statistical  gross  descriptors  (time  factors,  number  of  participants,  frequency  of 
responses,  length  of  threads,  and  the  like),  and  two  threads  examined  qualitatively  as  instances,  analyzed 
informally  in  an  attempt  to  ascertain  the  presence  of  characteristics  of  a community  of  practice  through  the 
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application  of  Wenger’s  theoretical  framework. 

An  interest  in  establishing  gross  trends  and  descriptors  implied  the  use  of  some  broad  categories  for  the 
contributions  to  the  list.  Marra  et  al  (2004)  examined  two  forum  discourse  analysis  frameworks  with  similar 
goals  - the  IAM  model  (Gunawardena,  Lowe,  & Anderson,  1997),  and  the  Newman,  Webb,  and  Cochrane 
model  (Newman,  Webb,  & Cochrane,  1996)  - but  these  methods,  more  focused  on  the  instructional  goals  of 
online  coursework,  were  viewed  as  too  general  to  be  useful  in  this  setting,  and  too  divorced  from  the  interests 
and  foci  of  the  Wenger  framework. 

Rather  than  simply  separating  thread  initiation  postings  from  responses,  postings  were  categorized  by  a 
somewhat  detailed  assessment  of  function: 

•S  Informational  (not  implying  or  requiring  response,  though  some  did), 

•S  Question, 

•S  Response  on  topic, 

•S  Response  off  topic  (“on”  or  “off”  was  determined  through  a simple  matching  of  subject  line  to 
content),  and 
■S  Other. 

This  framework  was  an  attempt  to  establish  a structural  relationship  between  postings  and  threads,  and 
to  supplement  the  natural  time  hierarchy  of  asynchronous  interaction. 

In  addition  to  disaggregating  and  reporting  by  subject  line,  contribution  content  was  categorized  by  general 
subject,  including 

•S  Instructional  (applying  to  the  classroom), 

•S  List  use  (postings  about  the  use  of  the  forum  itself), 

S Technical  (how-to  and  other  technology-specific  questions), 

•S  Policy  (having  to  do  with  school  or  district  level  technology  integration  policies  and  strategies), 

•S  Personal  (concerned  with  non-professional  interests), 

■S  Me  too  (one-line  postings  of  simple  agreement  or  thank-you’s),  and 
•f  Other. 

This  additional  framework,  it  was  hoped,  would  allow  the  data  to  paint  a broad  picture  of  how  the  list 
was  being  used  by  its  participants.  Through  the  use  of  both  of  these  frameworks,  it  was  hoped  that  a general 
characterization  of  the  interests  and  abilities  of  the  membership  would  arise,  as  well  as  the  presence  or  absence 
of  potential  community-building  interactivity. 

A detailed  examination  of  instances  of  or  threads  in  the  interaction  did  not  easily  lend  itself  to  the  use 
of  an  acutely  analytic  approach.  Over  the  past  three  decades,  a great  deal  of  attention  has  been  paid  to  the  use  of 
conversational  analysis  (CA)  to  describe  human  verbal  interaction,  and  there  are  implications  in  this  approach 
for  computer- mediated  communication  (Mazur,  2004).  However,  beyond  the  mediating  effects  of  the 
technology  itself,  there  are  factors  which  tend  to  limit  the  direct  application  of  CA  tools  in  this  context.  Such 
concepts  as  turn -taking,  repair,  and  adjacent  pairs  of  exchange  do  not  occur  naturally  in  a setting  where 
discussion  participation  takes  place  interspersed  between  other  work  responsibilities,  and  response  times  can 
often  be  measured  in  days.  As  we  shall  see,  the  emerging  character  of  the  list,  and  the  nature  of  the  interaction 
between  its  participants,  produced  a formality  which  more  closely  matches  correspondence  than  conversation. 

Gunawardena,  Lowe  et.  al  (1997)  gives  a review  of  several  analysis  models,  and  contributes  one  as 
well,  which  were  applied  to  an  on-line  discussion  occurring  over  a week.  However,  the  focus  of  the  studies 
cited  there  were  concerned  with  constructed  knowledge  and  learning  rather  than  an  assessment  of  community. 
Although  an  important  part  of  the  Wenger  model,  the  construction  of  knowledge  is  beyond  the  scope  of  this 
paper,  and  will  be  discussed  in  brief  in  “Implications  for  further  research.”  Hence,  for  the  purpose  of  analysis  in 
this  context,  two  threads  - one  exemplary,  one  perhaps  not  - were  simply  compared  to  the  Wenger  model,  and 
an  attempt  was  made  to  assess  the  presence  of  indicators  of  the  various  components  of  that  model. 

Research  Findings  and  Discussion 

Gross  Descriptors 

During  the  time  span  of  the  data,  the  86  members  of  FCPS-TRT  -L  produced  a total  of  164  postings  by 
43  members.  These  postings  represented  39  distinct  subject  lines,  with  a mean  thread  length  (postings  with  the 
same  subject  line)  of  4,  with  a standard  deviation  of  4.86  over  a range  of  1 to  27.  As  we  shall  see  in  our  posting 
function  data,  there  are  a high  number  (18)  of  informational  postings  which  did  not  imply  the  need  for  a 
response  (although  5 did  garnish  responses  anyway).  If  we  remove  the  remaining  unanswered  13  informational 
postings,  the  mean  thread  length  becomes  5.60  with  a standard  of  deviation  of  5.46,  a mode  of  3,  and  a range  of 
2 to  27.  Of  the  43  participants,  the  mean  number  of  contributions  from  each  was  3.42  with  a standard  deviation 
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of  5.68,  and  a median  and  mode  of  2 and  1,  respectively.  Since  I was  the  list’s  designer  and  moderator,  I 
contributed  a total  of  37  postings  to  the  list  - more  than  triple  the  next  most  active  contributor.  Dropping  this 
value  as  an  outlier  produced  a mean  of  2.61  with  a standard  deviation  of  2.22. 

In  an  attempt  to  ascertain  the  overall  activity  for  the  list  as  well  as  the  engagement  in  specific  topics, 
posting  response  times  are  reported  in  three  categories.  As  is  often  the  case,  the  median  time  proves  to  be  a 
better  descriptor  for  describing  the  overall  pace  of  the  participation,  since  there  were  occasional  long  gaps 
between  contributions  (the  maximum  was  6.67  days). 

Raw  Response  Times  (time  span  between  postings): 

■S  Mean  Response  Time:  9:40  hours 
S Median  Response  Time:  29.42  minutes 
1st  Response  to  Question  (time  between  a question,  and  the  first  posted  response): 

■S  Mean  Response  Time:  13:28  hours 
S Median  Response  Time:  2:28  hours 
Response  to  Thread  (time  between  thread  contributions): 

■S  Mean  Response  Time:  3:06  hours 
S Median  Response  Time:  26.35  minutes 

The  postings  were  categorized  as  to  their  function  within  the  discourse.  The  determination  of  “On 
Topic”  versus  “Off  Topic”  was  done  by  simple  inspection  - if  the  content  of  the  posting  matched  its  subject 
line,  it  was  considered  on  topic,  hence  many  of  the  “Off  Topic”  postings  may  actually  have  contributed  to  the 
thread  in  some  way,  or  have  been  “Me  Too”  or  personal  contributions. 

Total  postings  by  Type 

S Response  - On  Topic:  87 
•/  Response  - Off  Topic:  33 
•f  Question:  20 
>4  Informational:  18 
■S  Other:  6 

The  broad  topic  categories  proved  to  work  reasonably  well  - there  were  very  few  “fence  sitters”  - 
postings  which  could  have  fallen  into  more  than  one  category. 

Total  postings  by  Category 

■S  Technical:  44 

•S  Instructional:  40 

■S  Policy:  29 

•S  Personal:  15 

•S  List  Use:  21 

■S  “Me  too”/”Thank  You”:  13 

■S  Other:  2 

The  above  gross  descriptors  paint  a reasonable  picture  of  the  nature  of  the  contributions,  with 
implications  for  the  participants  themselves.  Within  the  first  two  days  of  the  list,  I posted  a “this  is  how  to  deal 
with  lots  of  email  from  a list”  remark,  which  defined  list  participation  as  strictly  voluntary: 

. . .you  have  permission  to  be  cutthroat!  If  you  don't  have  time,  or  your  email  box  is 

full,  just  delete  the  FCPS-TRT  -L  email  without  reading!  Nobody  will  know,  or  care! 

If  anyone  really  does  not  want  to  be  in  this  forum,  please  let  me  know,  and  I'll 

remove  you  - no  questions  asked... 

Despite  this,  half  of  the  members  of  the  list  chose  to  participate  during  the  data  time  span,  and  only  one 
person  asked  to  be  removed.  With  only  13  of  the  participants  contributing  once,  most  who  chose  to  participate 
did  so  a number  of  times  on  a range  of  subjects.  As  indicated  by  this  and  the  distribution  of  the  members’ 
contribution  counts,  one  can  clearly  see  that  this  list  - still  in  its  very  early  stages  of  development  - displays 
characteristics  of  mutual  engagement  beyond  the  structure  and  function  of  the  list  itself. 

The  response  time  data  paint  a picture  of  participation  which  belies  the  physical  positioning  of  this  list 
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in  the  professional  workday  of  its  participants.  The  overwhelming  majority  of  the  postings  (133  of  164)  took 
place  from  7:30  am  to  4:30  pm,  a reasonable  “normal  working  day”  range,  hence  most  postings  came  from 
members  who  were  mixing  their  participation  into  their  other  responsibilities.  This  makes  the  median  raw 
response  time  and  median  response  to  thread  time  quite  remarkable. 

The  list  can  be  characterized  by  a fairly  formal  and  business-like  mode  of  participation.  Off-topic 
contributions  to  threads  are  the  exception  rather  than  the  rule,  and  personal  or  simple  “Me  too”  contributions 
constitute  a very  small  percentage  of  the  overall  traffic.  In  addition,  the  use  of  a color-coded  representation  of 
the  threads  (see  Appendix  A)  as  they  evolved  on  the  list  shows  that  the  scattered  nature  of  thread  connections 
and  adjacency  of  thread  contributions  gradually  formalized  over  time,  making  adjacency  of  contributions  only 
interesting  as  they  continued  the  thread.  This  reflects  the  substantial  difference  between  this  style  of  online 
forum,  and  synchronous  or  highly  active  forums  in  which  adjacency  contributes  to  the  overall  character  of  the 
discourse  (Herring,  1999).  Hence  an  examination  of  this  aspect  of  the  data  was  discarded  as  uninteresting. 

TRTs  often  worry  that  their  working  day  will  be  filled  with  simple  technical  support  issues,  but  with 
combined  posting  totals  for  “Instructional,”  “Policy,”  and  “List  use”  subjects  over  90  (well  over  half,  with 
purely  technical  postings  less  than  half  that),  it  is  clear  that  many  of  the  participants  of  this  list  are  seriously 
engaged  in  the  defined  purpose  of  the  forum.  Two  factors  tended  to  encourage  the  formality  of  this  particular 
list  community  - the  presence  of  a vocal  moderator  with  a specific  agenda  and  a large  number  of  postings,  and 
an  early  departure  of  a list  member  who  complained  of  a high  number  of  “chatty”  personal  postings  on  the  first 
days  of  the  list.  Nevertheless,  over  the  span  of  the  dataset,  the  subject  data  seem  to  reflect  the  sense  of  joint 
enterprise  through  the  reflective  process  of  negotiation,  an  important  part  of  the  Wenger  model. 

Thread  Instances 

For  the  purpose  of  this  analysis,  two  threads  will  be  examined:  The  Digital  Storytelling  thread 
(Appendix  B)  and  the  Mobile  Lab  thread  (Appendix  C).  The  former  is  the  second  longest  thread  in  the  dataset 
(16  contributions),  and  serves  as  an  exemplary  exchange  on  one  of  the  district’s  classroom  technology 
integration  initiatives.  The  latter,  in  contrast,  actually  comprises  two  threads.  The  original  poster  attempted  to 
restart  the  thread  with  a second  asking  of  essentially  the  same  question.  It  had  a total  of  1 1 postings  - 4 in  the 
first  group,  7 in  the  second.  Beyond  this,  there  has  been  no  attempt  to  quantify  the  nature  of  these  postings.  The 
analysis  is  merely  descriptive  - the  nature  of  the  contributions,  and  the  overall  character  of  each  thread,  are 
described  as  they  potentially  reflect  the  Wenger  model  of  a community  of  practice. 

Digital  Storytelling:  The  thread  concerns  itself  with  an  instructional  topic  - the  expansion  of  tools  used 
in  the  process  of  rendering  personal  narratives  (or  other  writing)  as  electronic  multimedia.  Initiated  by  a 
technology  resource  teacher  who  was  the  coordinator  of  this  initiative,  the  question  mentions  the  current 
technology  being  used  (a  video  editing  suite  and  digital  imaging  software),  and  wonders  if  any  others  are  being 
tried.  The  responses  occur  in  a solid  time  block  over  a four-hour  period,  with  9 list  members  contributing.  The 
16  contributions  include  three  off-topic  postings,  all  in  reference  to  clerical  errors  in  the  posting  process. 

The  contributions  exhibit  a great  deal  of  interest  and  thoughtful  participation.  Although  most  of  the 
contributors  were  attendees  of  a face-to-face  professional  development  class  on  this  subject,  some  were  not.  The 
postings  are  characterized  by  a problem-solving  and  divergent-thinking  approach  to  the  question,  and  exhibit 
directly  a reflective  process  of  negotiation  with  mutual  accountability,  as  the  contributors  examine  each  other’s 
ideas  or  volunteer  past  successes . . . 

Joy,  (and  anyone  else  interested)  The  "talking  books”  I made  were  created  with 
Microsoft  Publisher.  It's  a really  easy  way  to  make  a story  with  sounds.  The  info, 
about  the  PD  I did  is  at: 

http://dixie.fcps.net/Professional_Development/Making_Talking_Books.htm  You 
can  also  use  Front  Page  but  I think  Publisher  is  a little  easier.  It  would  be  possible  to 
do  with  Power  Point  as  well.  You  are  only  limited  by  your  imagination,  (and  time, 
of  course,  always  a problem!)  Jan 

The  contributions  also  clearly  illustrate  the  fact  that  this  forum  serves  to  display  complementary  participation 
and  provides  for  specialization,  as  the  varieties  of  the  participants’  expertise  and  interest  unfold.  Although  it  is 
difficult  to  ascertain  how  much  of  the  character  of  the  postings  is  a function  of  the  medium,  its  defined  role,  and 
the  formal  nature  of  the  relationship  between  the  participants,  there  is  evidence  that  the  interaction  actually 
represents  a community  of  practice  (the  following  contribution  follows  one  in  which  the  participant  failed  to 
identify  himself  with  an  auto-signature)... 
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Nathan:  You're  on  here  as  a courtesy. ..you  better  shape  up!  ;-) 

Assessing  the  nature  of  the  “...shared  repertoire....  resulting  from  engagement”  is  difficult  to  assess.  Obviously, 
it  is  not  possible  for  a virtual  environment  to  provide  visibility  for  the  results  of  the  community  in  action,  since 
the  members  of  the  list,  by  and  large,  are  distributed  one  each  to  over  50  school  sites.  The  actual  repertoire  of 
actions  as  they  are  made  manifest  in  the  classroom  will  not  appear  in  the  forum,  except  as  they  are  described 
(Jan’s  example  is  one  of  several  that  does  so).  Nevertheless,  the  entire  thread  on  its  face  implies  the 
development  of  such  a repertoire. 

Mobil  Labs:  This  thread,  initiated  by  a district  TRT  who  was  doing  some  exploration  for  a conference 
presentation,  became  a brief  discussion  primarily  between  two  individuals  - the  asker  of  the  question,  and  one 
elementary  school  participant.  The  low  overall  level  of  participation  in  this  thread  may  be  evidence  of  the 
weakness  of  the  community  in  supporting  all  discussed  topics.  It  might  also  be  evidence  of  the  specialized 
nature  of  the  topic  and  the  relatively  rare  use  of  the  technology  being  discussed.  It  is  impossible  to  determine 
this  from  the  dataset. 

The  first  instance  of  the  thread  included  two  contributions  from  one  participant,  with  a second  expanding  on  the 
first’s  remarks  (he  was  a floating  TRT  who  included  the  same  school  in  his  rotation).  Since  the  thread  promptly 
died,  it  was  reintroduced  three  days  later  by  the  original  asker.  The  second  instance  of  the  thread  produced  two 
more  postings  from  the  first  respondent,  plus  one  new  participant’ s contribution,  plus  two  short  remarks  (one  of 
which  was  off  topic)  and  a further  remark  by  the  asker. 

This  thread  was  selected  because  of  its  contrast  to  the  first,  that  is,  the  inability  of  this  asker  to  extract  a wide 
participation  in  the  exchange.  This  illustrates  one  of  the  problems  inherent  in  applying  a theoretical  definition  of 
a hoped-for  goal  as  a tool  for  analyzing  the  success  or  failure  of  electronic  discourse  in  meeting  or  illustrating 
that  goal.  Obviously,  the  low  participation  level  of  the  exchange  means  that  the  sense  of  community, 
specifically  “complementary  participation. . .as  an  extension  of  practice,”  is  not  in  evidence  for  the  group  as  a 
whole.  However,  the  nature  of  the  individual  contributions  shows  an  insight  and  participation  in  the  topic  (and, 
by  extension,  the  forum  itself)  which  establish  a joint  enterprise  and  shared  repertoire... 

It's  much  cheaper  too.  The  cart  is  available  through  Alpha  Smart  for  about  $550.00 

vs.  $5000  or  more  for  the  big  cart.  It  has  some  drawbacks  to  be  sure,  but  is  very 

functional.  I can  send  pics  if  anyone  is  interested... 

Overall,  the  on-task  contributions  were  very  much  longer  than  the  norm  for  this  forum  -another 
potential  explanation  for  the  small  number  of  contributions,  since  large  postings  often  intimidate  or  overwhelm 
members  who  have  insufficient  time  for  lengthy  thoughtful  exchanges.  This  meant  that,  in  several  cases,  a 
variety  of  scenarios,  problems,  and  insights  were  included  in  a single  posting,  making  the  contribution  to  the 
community  that  much  potentially  stronger. . . 

Kim,  Today  was  the  first  day  with  the  wireless  lab  at  SMS.  A 7th  grade  social 
studies  class  was  the  first  to  use  the  new  wonders  of  technology.  One  problem  we 
encountered  was  that  we  did  not  have  enough  computers  with  administration.  Middle  School 
Review  , and  a teacher  having  checked  one  out  we  were  a few  short  for  everyone  to  have  one. 

The  first  activity  was  in  the  library  which  worked  well  with  more  space  and  large  tables.  I 
would  encourage  anyone  who  is  considering  this  venture  to  have  a in-service  for  the  staff  in 
utilizing  the  technology. 

As  in  the  Digital  Storytelling  thread,  it  is  difficult  to  assess  how  much  of  the  formality  and  “all 
business”  nature  of  the  postings  are  an  extension  of  the  structure  of  the  medium,  but  it  seems  unlikely.  Again, 
there  were  off-topic  exchanges  which  showed  that  the  structure  provided  for  personal  banter. . . 

Great  suggestion,  so.. .John.... when  do  you  want  us  to  come  to  SMS  for  your  PD  session? 

If  I knew  who,  I could  then  invite! 


This,  again,  was  an  exchange  born  of  the  lack  of  an  identifying  signature  - when  this  technical 
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problem  was  corrected  through  an  improvement  in  the  structure  of  the  list,  the  banter  didn’t  disappear.  But  the 
personal  exchanges  and  collegiality  associated  with  many  online  discussions  remained  at  a relatively  low  level 
of  occurrence. 

There  are  several  other  threads  in  the  dataset  which  serve  to  illustrate  the  cohesion,  participation  level, 
and  other  evidence  of  the  possibility  of  a community  of  practice  within  the  members.  Especially  revealing  is  the 
longest  of  the  threads  at  27  contributions  on  student  use  of  teacher  workstations  in  the  classroom.  This  thread 
lead  directly  to  a further  discussion  of  computer  workstation  security,  and  resulted  in  the  preparation  and 
scheduling  of  a professional  development  class  on  the  topic.  Clearly  this  is  evidence  of  the  mutual  engagement, 
the  joint  enterprise,  and  the  shared  repertoire  of  a potential  community  of  practice.  However,  in  informal 
discussion  with  some  of  the  members  of  the  list,  the  lackof  face-to-face  discussion  was  cited  as  a concern.  Of 
course,  the  irony  of  these  observations  in  the  face  of  the  previous  year’s  lack  of  participation  in  meetings  is  not 
to  be  missed.  But,  overall,  one  might  conclude  that  a great  deal  of  the  benefits  of  a community  can  be  obtained 
by  this  text -based  medium,  and  this  evidence  can  be,  to  a certain  extent,  derived  directly  from  the  text  itself. 

Implications  for  Design,  and  Further  Research 

With  a goal  of  establishing  a community  of  practice  through  the  use  of  online  discourse,  and  assessing 
its  veracity  through  an  analysis  of  the  resultant  archive  logs,  there  are  several  limitations  inherent  in  this 
research  design.  First  of  all,  a more  direct  marriage  of  analysis  and  model  is  required,  since  a simple  inspection 
for  evidence  leaves  much  to  be  desired.  Also,  a more  formal  delineation  of  the  discourse  using  more  general 
tools  of  analysis  might  give  a better,  more  transferable  method  of  assessing  the  ability  of  this  medium  in 
meeting  its  stated  goals,  and  in  describing  its  character  in  a way  more  easily  compared  to  other  such 
communities.  As  mentioned  previously,  the  Wenger  model  directly  addresses  the  presence  and  importance  of 
learning  as  an  integrated  part  of  the  community  of  practice  model,  placing  it  at  the  center  of  practice, 
community,  identity,  and  meaning  (Wenger,  1998,  5).  Assessing  the  process  of  learning  in  corrputer  mediated 
communication  has  a more  extensive  history  of  practice  (Gunawardena  et  ah,  1997),  and  the  use  of  those  tools 
in  this  study  might  have  served  to  increase  the  accuracy  of  the  community’s  description. 

Another  problem  with  the  scope  of  the  interests  of  a community  as  compared  to  a text  -only  dataset  is 
the  inability  of  the  data  to  represent  all  of  the  membership.  Although  the  participation  level  in  this  forum  was 
really  quite  high,  there  is  no  way  to  extract  from  the  dataset  the  nature  of  wow-participation,  that  is,  why 
members  failed  to  post  or  react  to  postings.  And,  as  Wenger  and  Wenger  and  Lave  have  noted  (Wenger,  1998, 
p.  149),  legitimate  peripheral  participation  - that  is,  engaging  in  community  activity  in  various  stages,  from  the 
outside  to  within,  is  highly  characteristic  of  novices  who  are  in  the  process  of  establishing  membership  in  a 
community.  An  informal  discussion  with  a list  member  revealed  that  she  had  information  relevant  to  one  of  the 
threads,  but  did  not  bother  to  pass  that  information  on  to  other  members.  It  would  be  quite  revealing  to  attempt 
to  discover  if  non-participant  members  were  “lurking”  (watching  without  participating,  as  this  member  was)  or 
simply  taking  the  advice  I offered  at  the  inception  of  the  forum  and  deleting  contributions  without  reading  them, 
for  reasons  of  time  or  other  factors.  One  cannot  realistically  hope  for  100%  participation  in  an  online  forum 
(many  large  forums  are  lucky  to  enjoy  5%  or  more),  but  the  Wenger  model  attempts  to  describe  the  entire 
community  rather  than  only  the  most  visible  and  vocal  members  of  it.  An  attempt  was  made  to  get  at  some  of 
this  information  through  an  informal  online  survey,  participation  in  which  was  solicited  through  the  list. 
However,  the  survey  enjoyed  even  lower  participation  than  the  forum  itself  - not  surprisingly,  since  the 
participants  in  the  survey  were  most  likely  to  be  participants  in  the  forum,  which  is  already  a subset  of  the 
whole  membership.  Hence  the  expansion  of  the  research  that  these  interests  imply  must,  by  definition,  take  one 
away  from  the  capabilities  of  the  forum  itself,  and  would  require  a great  deal  more  effort  than  a simple  analysis 
of  text. 

One  of  the  difficulties  with  the  numerical  data,  the  “gross  descriptors,”  was  a lack  of  data  with  which 
to  compare.  The  overwhelming  majority  of  the  small  research  data  on  electronic  discourse  was  on  environments 
created  to  enhance  the  interactive  nature  of  online  courses,  and  most  of  the  analysis  was  at  the  “micro”  level 
(CA-styled  analysis)  or  the  “macro”  level  (overall  effectiveness  in  enhancing  class  viability  and  achieving 
learning  goals).  Those  few  examples  of  research  which  were  aimed  at  community  building  amongst  teachers 
gave  no  numeric  data  at  all.  Although  every  forum  is  different,  without  some  purely  statistical  representation  of 
participation  it  is  impossible  to  establish  expectations  and  norms  for  response,  in  the  goal  of  painting  a picture 
of  the  success  or  failure  of  a forum  in  enhancing  the  goals  and  interests  of  the  community  which  it  represents. 

The  final  concern,  of  course,  is  for  the  encouragement  of  community  itself.  The  application  of  the 
Wenger  community  of  practice  model  to  teachers  is  not  without  precedence  (Crawford,  2001;  Davenport, 

2001),  and  it  has  great  implications  for  a profession  struggling  under  the  burden  of  assessment,  accountability, 
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and  classroom  practice  pressures.  The  interests  of  technology  integration  are  diverse,  and  the  front-liners  in  this 
battle  are  often  disconnected  and  displaced.  The  need  for  community  is  strong,  and  the  exploration  and 
expansion  of  computer  mediated  communication  in  this  support  seems  full  of  potential.  This  line  of  thinking  has 
strong  implications  for  the  design  of  electronic  forums.  Wenger  gives  some  direction  on  this  point,  noting  that 
communities  of  practice  are  “as  old  as  humankind  and  existed  long  before  we  started  to  concern  ourselves  with 
systematic  design  for  learning.  Communities  already  exist  throughout  our  societies — inside  and  across 
organizations,  schools  and  families... some  are  potential... some  are  active,  some  are  latent(1998,  p.  228).”  The 
data  and  analysis  of  online  talk  presented  in  this  preliminary  study  gives  needed  information  on  how  the  content 
and  structure  of  the  interaction  can  impede  or  promote  the  progress  from  a potential  to  an  active  community  of 
practice. 
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Contributions  to  These  Practices  from  a Professional  Program 

Ugur  Kale 
Indiana  University 

Introduction 

Teacher  professional  development  is  assumed  to  be  enhanced  through  reflective,  collective, 
collaborative  professional  communities  (Little,  2003). Universities  in  these  teacher  community  developments 
play  a big  role  by  providing  teachers  with  pedagogical  helps  and  professional  developments.  By  not  only 
encouraging  teachers  to  go  to  conferences,  do  research  publications,  and  use  technology  in  their  teaching 
practices  but  also  leading  them  to  work  together,  evaluate  each  others’  instruction,  and  do  collaborative  lesson 
plans,  these  professional  university  driven  communities  seem  to  remarkably  contribute  to  teachers’s  teaching 
practices  and  professional  development. 

Getting  to  know  how  these  communities  influence  the  teaching  practices  through  analyses  of  teaching 
applications  in  classrooms  become  an  important  issue  not  only  for  the  communities  as  a way  of  assessing  their 
programs  but  also  for  many  practitioners  and  professionals  in  education  who  would  like  to  see  possible  teaching 
situations,  issues,  teaching  strategies. 

As  one  group  of  these  practitioners  and  professionals,  instructional  designers  and  instructional  theory 
developers  are  aware  of  the  importance  of  the  possible  situationalities  for  teaching  practices.  It  is  because  one  of 
the  main  goals  for  these  professionals  is  to  describe  how  instruction  should  look  like,  what  possible  conditions 
and  issues  may  take  place  will  be,  and  so  on  within  instruction.  Unlike  learning  theories  that  describe  how 
learning  occurs,  instructional  theories  are  more  goal-oriented  and  tend  to  give  more  explicit  ways  to  follow  in 
supporting  learning  in  instruction  (Reigeluth,  1999).  In  this  respect,  instructional  theories  are  expected  to 
provide  methods  in  details  and  variety  of  situations.  In  that  way,  audiences  for  the  theories  who  are  teachers, 
instructional  design  and  theory  developers,  and  other  practitioners  interested  in  pedagogical  issues,  can  make 
sense  of  and  apply  these  prescriptive  theories  into  their  practices  successfully. 

The  case  study  presented  in  this  paper  aims  to  document  the  teaching  practices  of  four  high  school 
mathematics  teachers  who  have  been  involved  in  a math  professional  development  program  provided  by  one  of 
the  Midwest  Universities.  This  paper  aims  at  benefiting  the  professional  development  program  and  many 
educators,  such  as  instructional  design  and  theory  developers  by  laying  out  the  teachers’  teaching  practices  and 
examining  influence  of  the  program  on  these  practices. 

In  order  to  analyze  teaching  practices,  I thought  it  is  important  to  find  a well-accepted  framework  so 
that  laying  out  the  individual  aspects  of  the  teachers’  instruction  becomes  easier  and  meaningful.  For  this  study, 

I chose  Gagne’s  nine  events  of  instructions  as  the  framework  because  as  accepted  by  many  professionals, 
Gagne’s  events  are  assumed  to  be  the  most  general  events  that  can  take  place  in  any  type  of  learning  situations. 
For  a better  learning,  these  events  should  be  accomplished  to  satisfy  the  necessary  learning  conditions  for 
relevant  learning  type  (Aronson,  1983)  such  as  motor  skills,  intellectual  skills,  attitudes,  cognitive  strategy  and 
so  forth.  Even  though  “Gagne’s  nine  events  of  instruction  are  his  methods  of  instruction’’  (Reigeluth,  2003), 
these  general  events  -methods  hardly  mention  the  situations  where  the  methods  should  be  applied  in  different 
ways.  That’s  why,  any  situational  differentiation  that  could  be  found  in  the  teachers’  teaching  practices  may 
provide  details  to  applying  these  nine  events  under  many  conditions,  which  Gagne  gives  little  guidance  about. 
Below  are  the  nine  events  that  Gagne  proposes: 

Event  1-  Gain  the  Learner’s  Attention:  This  event  includes  techniques  to  appeal  learners’  curiosity  and 
interest  such  as  presenting  something  novel,  giving  challenging  situations  or  problems,  putting  students  in 
uncertainty,  and  so  forth. 

Event  2-  Inform  the  Learner  of  Objective:  This  is  about  familiarizing  students  with  what  they  will  be 
learning  during  the  lesson  and  how  they  will  be  assessed.  Knowing  what  to  learn  provide  students  with  a 
determined  focus  in  their  learning. 

Event  3-  Stimulate  Recall  of  Prerequisite  Learning:  This  is  to  help  students  to  recall  their  necessary 
prior  knowledge  and  experiences  to  understand  the  new  acquired  information  in  a meaningful  way. 

Event  4-  Present  Stimulus  Material:  It  is  the  presentation  of  the  content  through  variety  of  media  and 
approaches  to  expose  information  to  the  learner. 
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Event  5 - Provide  Learning  Guidance:  This  is  about  the  methods  such  as  providing  examples,  hints, 
cognitive  tools  - handouts  to  make  learning  easier. 

Event  6-  Elicit  Performance:  This  event  is  about  providing  opportunities  to  students  for  practicing 
through  having  them  “do”  instead  of  having  them  just  “listen”. 

Event  7-  Provide  Feedback  about  Performance  Correctness:  This  includes  methods  in  which  learners 
are  given  a description  of  how  they  have  done  in  given  practices.  It  could  be  verbal,  written,  or  other  forms. 

Event  8-  Assess  Performance:  It  is  more  about  a process  of  assessing  students’  performance  after  they 
have  been  exposed  to  a learning  event. 

Event  9-  Enhance  Retention  and  Transfer:  This  is  the  event  where  learners  are  given  more 
opportunities  to  practice  or  review  the  materials  so  that  they  can  recall  them  in  a later  time  easily  and 
meaningfully. 


Method 

This  study  is  a case  study  in  which  a group  of  teachers’  teaching  practices  and  professional 
development  experiences  were  examined  in  a bonded  system,  which  is  a professional  development  community 
that  affects  these  practices  and  experiences.  As  Merriam  mentions  (1998),  in  case  studies,  researchers  are  more 
interested  in  discovering  and  interpreting  based  on  a phenomena  taking  place  in  a bounded  place,  instead  of 
testing  a hypothesis.  The  teachers’  teaching  experiences  within  their  professional  development  process  is  the 
phenomenon  that  I am  trying  to  “discover”  through  this  study.  The  approach  I took  for  the  study  could  be 
hermeneutic  (Brantlinger,  1993)  because  I try  to  see  the  teachers’  practices  from  their  “eye”  along  with  myself 
interpreting  what  they  “see”. 


Setting  & Participants 

Participants  were  four  high  school  mathematics  teachers  that  have  five,  thirteen,  sixteen,  and  twenty- 
eight  years  teaching  experiences  respectively.  They  were  currently  participating  in  a professional  development 
program  provided  by  one  of  the  Midwestern  Universities.  This  math  teacher  professional  development  program 
aims  at  engaging  many  practitioners,  secondary  and  high  school  math  teachers,  university  mathematicians  and 
mathematics  educators  in  a collaborative  environment  where  mathematics  teachers  can  get  benefit  for  their 
teaching  practices  and  professional  developments. 

The  teachers  involved  in  this  study  were  currently  teaching  algebra,  geometry,  and  calculus,  to  9-12 
graders  by  the  time  study  was  being  conducted.  While  the  teacher-George  with  a twenty  eight  and  Kurt  with 
thirteen  years  teaching  experiences  had  been  involved  in  the  meetings  of  the  math  professional  development 
program  for  more  than  2 years,  the  teacher-Kristen  with  5 years  teaching  experience  recently  got  involved  in 
meetings  for  last  couple  of  months.  On  the  other  hand,  while  the  other  teacher-John  with  sixteen  years  teaching 
experience  had  been  in  the  professional  development  program  for  2 years,  he  had  not  been  able  to  attend  the 
meetings  regularly  because  of  his  busy  schedule.  The  teachers  were  all  from  a same  high  school  in  one  of  the 
midwestern  city  that  is  dominated  by  a big  university  by  which  the  professional  development  program  has  been 
driven. 


Researcher 

Pursuing  a doctoral  degree  in  instructional  technology  department  at  a school  where  the  professional 
development  program  had  been  carried  out  made  it  easier  for  me  to  contact  with  the  manager  and  the  director  of 
the  math  teacher  development  program.  After  initial  contact  and  agreement  based  on  my  research  purpose  and 
interest  in  instructional  theory  development  and  possible  benefits  that  the  program  can  get  from  the  study,  I was 
allowed  to  ask  the  teachers  to  participate  in  the  study.  I was  introduced  to  the  teachers  as  a graduate  student 
member  doing  research  studies  for  the  program. 

Data  Collection  and  Analysis 

The  data  collected  are  mainly  through  interviews.  In  addition,  observations  of  initial  meetings  and 
document  analysis  regarding  the  mission  and  the  structure  of  the  program  were  used  as  ways  of  collecting  data. 

Document  Analysis 

Mainly,  I searched  in  the  website  of  the  program  for  its  mission,  activities,  regular  meetings  and  so 
forth.  It  just  gave  me  an  initial  plan  for  meeting  the  director  and  the  manager  of  the  program  and  proposing  my 
research  agenda. 
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Observation 

To  be  introduced  to  the  teachers  in  the  program,  I participated  in  one  of  the  regular  meetings  taking 
place  within  the  program.  This  gave  me  a chance  to  see  the  structure  of  meetings  and  issues  held  during  the 
meetings  as  well  as  an  opportunity  to  meet  the  teachers  face  to  face. 

Interview 

Convenience  sampling  approach  was  used  to  involve  teachers  in  the  study  because  by  the  time  study 
was  started,  it  was  getting  close  to  the  second  half  of  the  first  semester  in  high  schools  when  many  teachers  are 
pretty  busy  their  schedules  for  the  school. 

One  or  two  weeks  after  the  meeting  I attended,  teachers  were  emailed  by  the  manager  of  the  program 
whether  they  would  be  willing  to  participate  in  a proposed  study  for  which  I would  be  interviewing  them. 
Among  them,  did  four  teachers  agree  to  participate.  All  of  the  teachers  were  (45  minutes  to  1 hour)  interviewed 
in  their  schools  within  the  following  2 weeks. 

The  questions  to  be  asked  in  the  interviews  had  three  different  facets:  The  first  one  included  general 
questions  such  as  the  teachers’  years  of  teaching  experiences,  how  they  had  got  involved  in  the  program,  what 
their  initial  expectations  were  and  so  forth.  The  second  one  was  about  teachers’  teaching  practices  in  details. 
Gagne’s  nine  events  were  used  as  a framework  regarding  what  tactics  teachers  are  using,  and  what  they  are 
encountering  in  each  event.  The  last  part  was  focused  on  how  the  teacher  professional  development  program 
had  been  contributing  to  their  teaching  practices  and  what  their  possible  suggestions  would  be  regarding  a better 
professional  program.  Each  teacher’s  interview  was  audio  taped  and  transcribed  for  the  analysis. 

Findings 


The  Teachers  and  the  Program 

Teachers  were  mainly  participating  in  the  program  by  attending  two  main  meetings  held  in  the 
university;  (1)  joint  meetings  in  which  teachers,  university  mathematicians  and  mathematic  educators  gathered 
together  to  discuss  the  future  activities,  goals,  and  directions  for  the  program.  Approximately  40  joint  meetings 
had  been  held  since  2000,  with  an  average  of  5 meetings  per  each  semester  (including  summers).  (2)  The  other 
meetings  were  those  in  which  teachers  meet  as  subgroups  regularly  in  every  two  weeks  to  discuss  about  a 
particular  lesson  plan  they  are  developing  consistent  with  their  particular  teaching  area.  While  Kurt  and  Kristin 
were  in  a group  that  was  dealing  with  multiple  representations,  George  was  in  a group  for  proof  in  math,  and 
John  was  in  a group  for  technology  in  math  area. 

Based  on  my  observation  and  information  gathered  from  the  website,  in  the  second  type  of  meetings, 
which  are  called  LSG  (lesson  study  groups)  meetings,  teachers  generally  spent  30  minutes  for  a general 
discussion  about  the  next  week’s  agenda.  Then,  they  grouped  in  their  LSG  groups-  multiple  representation, 
proof,  or  technology  groups,  etc.  to  elaborate  a lesson  plan  that  they  have  been  trying  to  form  for  long  time. 
Their  problems  in  their  teaching  experiences  could  also  be  a discussion  topic  if  other  teachers  in  the  group  were 
also  interested  in  it.  For  example,  when  elaborating  the  lesson  plan  in  proof  group,  one  of  the  teachers,  George, 
was  addressing  an  issue  that  he  considered  to  be  important  for  his  students  - the  lack  of  language  level  in  math 
and,  then,  others  continued  the  conversation  by  giving  their  own  examples.  This  similar  conversation  seemed  to 
show  me  that  there  could  be  some  problems  in  teachers  teaching  practices  that  I should  consider  in  my 
interviews  later  in  the  study.  That  definitely  led  me  to  put  an  additional  question  into  the  interview  questions 
regarding  what  kind  of  problems  that  teacher  can  face  in  their  teaching.  In  the  following  section,  I will  describe 
my  main  findings  from  the  interviews  of  the  teachers. 

Why  Are  the  Teachers  in  the  Program? 

When  asked,  the  teachers  seemed  to  have  similar  reasons  for  joining  the  program.  Looking  at  lessons 
in  depths,  trying  to  be  reflective  on  their  own  teaching  practices,  collaborating  with  other  teachers,  and  doing 
pedagogical  readings  were  the  general  reasons  why  the  teachers  got  involved  in  the  project.  On  the  other  hand, 
as  a professional  development,  the  program  seemed  to  address  other  expectations  of  the  teachers  such  as  being 
given  opportunity  to  go  to  conferences,  do  academic  publications,  and  use  pedagogical  resources  and 
technological  equipments  that  the  program  can  provide. 

What  Do  the  Teachers’  Teaching  Practices  Look  Like? 

The  followings  are  the  main  teaching  events  and  issues  that  the  teachers  had  interesting  points  and 
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strategies  about.  Thus,  I do  not  necessarily  mention  about  each  of  the  nine  events  here.  Rather,  in  the  conclusion 
part,  I put  a table  that  describes  all  of  the  activities  that  the  teachers  carry  for  each  event  as  a summary. 

Gaining  Students’  Attention:  The  teachers  mentioned  questioning,  making  announcements  about  the  lesson 
plan/objectives,  starting  with  high  level  of  tasks,  or  presenting  a problem,  as  the  main  ways  for  grabbing 
students’  attention.  In  many  instructional  tactics  challenging  students  with  questions  is  one  way  to  get  students 
attention  as  seen  in  teachers’  explanations  (Dalton,  2003).  However,  one  of  the  teachers,  Kristin,  seemed  to  be 
careful  regarding  presenting  challenging  question. 

In  the  past,  I tried  to  do  things  like  here  is  an  attention-getting  problem  “here  you  work  here  and  I work”.  It 
was  kinda  frustrating. 

In  many  learning  situations  if  students  are  left  alone  in  a given  challenging  question,  they  may  become 
frustrated.  It  is  a concern  that  both  Kristin  and  Kurt  pointed  out.  Regarding  challenging  question,  Kurt 
continued  with  another  point,  student  achievement  level: 

They  should  have  a background  to  figure  out  when  doing  problems.  I do  not  challenge  my  lower  level 

students. 

In  a similar  way,  George  mentions  about  challenging  students  in  relation  to  their  levels: 

I can  say  it  is  hard,  I have  a couple  of  students  in  second  year  cal[culus]  that  are  really  difficult. ...  To  challenge 
them  plus  keep  the  class  going  is  tough,  because  I can  develop  my  class  period  if  you  are  doing  things  that  they 
are  capable  of  doing  and  that’s  kinda  hard.  It  is  tough  to  have  the  appropriate  thing  for  every  student. 

As  seen  in  George  and  Kurt  explanations,  students  level  could  be  a problematic  situation  in  which  low 
level  of  student  are  more  likely  to  get  stressed  when  confronted  with  a challenging  problem.  On  the  other  hand, 
high  level  students  are  believed  to  get  benefit  from  it.  As  George  noted,  it  could  be  really  difficult  to  address 
issues  tailored  to  every  student.  Especially  when  the  class  size  is  considered,  posing  challenging  questions  to  the 
students  might  not  be  a preferred  option  for  teachers.  Hargreaves  and  her  colleagues  (1997),  examining  the 
class  size  and  student-teacher  interaction,  observed  that  challenging  questions,  which  students  need  to  give 
more  reasons  to  answer,  are  often  used  in  smaller  classes.  The  bigger  the  class  size,  the  more  likely  it  is  to  have 
different  level  of  students  and  the  less  often  teachers  ask  challenging  questions.  The  ideal  way  for  such  as 
situation  like  this  could  be  to  try  to  find  out  a middle  point  of  “how  often’’  to  use  challenging  questions  or  to 
figure  out  other  ways. 

Rather  than  starting  with  a question,  Kristin  suggested  another  way,  which  is  not  directly  targeted  on 
gathering  attention  but  on  keeping  students’  focus  in  a meaningful  way. 

I have  a folder  for  them,  they  go  and  check  that  so  they  are  busy  organizing  their  binders  firstly,  they 
are  required  to  keep  a binder  and  it  is  separated  into  sections  like  handouts  and  notes  homework  and 
quizzes  and  tests,  they  have  these  sections  and  they  go  to  their  folder,  you  know,  they  have  things  to  do,  they 
know  that  that  is  expected  so  they  bring  it  back  and  sit  down  and  organize  it  and  whatever  it  says,  they  need  to 
do  that  point  too. 

Keeping  students  busy  in  doing  some  sort  of  tasks  was  seen  as  another  way  of  maintaining  the  students 
focus  by  Kristin.  Maybe,  this  is  not  something  that  grabs  students’  attention  directly  but  something  that  warms 
them  up  before  the  class  starts.  On  the  other  hand,  this  is  not  to  say  that  starting  with  an  interesting  and 
challenging  question  will  always  cause  to  frustration  among  the  students.  As  long  as  students  are  given  chance 
to  think  and  discuss  about  the  question,  it  may  be  still  possible  to  stimulate  students  ’ thinking.  Those  students 
who  can  be  stressed  by  teachers’  challenging  question  may  have  more  effective  discussion  if  they  are  given 
chance  to  collaborate  with  each  other.  Maybe,  putting  students  in  pairs  and  letting  them  discuss  about  the 
question  will  work  well  in  leading  students  to  have  focus  and  pay  attention  to  the  class  flow  as  another  teacher, 
George,  did.  Talking  about  how  he  kept  students  attention  level  high,  George  went  on: 

I actually  question  them  to  ask  their  partners  to  respond  their  partner  or  “ here  is  something  I would 
like  you  to  work  on  for  a few  minutes’’  so  they  do  not  have  long  periods  of  times  whether  or  not  doing 
something. 

But  again,  without  enough  help  and  resources,  students  even  in  groups  can  feel  frustrated.  Students 
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confronted  with  bunch  of  information  and  problems  to  solve  without  any  adequate  resource,  can  be  in  trouble. 
Kristin,  trying  to  describe  the  ideal  challenge,  addressed  this  issue: 

To  me,  I guess  the  ideal  challenge  is  not  just  pushing  information  on  them  but  helping  them  learn  the 
information.  I never  ask  them  to,  “ok  put  away  your  book  and  do  not  look  at  that”  [instead]  “Use  all  your 
resources”  you  know  and  to  set  up  a challenge  in  that  way  that  you  [students]  always  have  resources  to  use.  So, 
“use  what  you  have”  and  “ use  somebody  sitting  next  to  you,  ask  some  questions  you  do  not  understand”. 

It  seemed  to  Kristen  that  attention  gathering  is  not  necessarily  to  ask  students  something  interesting 
and  challenging  that  directly  grabs  their  attention,  but  to  keep  students  focus  in  a meaningful  way,  which  could 
be  in  the  form  of  asking  students  to  organize  their  folders,  discuss  in  their  groups  or  pairs,  or  so  forth. 

Stimulating  Recall  of  Prior  learning  In  many  subject  matters,  understanding  a concept  is  very  important 
learning  outcome  like  in  mathematics.  Understanding  concept  is  not  just  to  describe  something  as  a concept  but 
rather  to  see  the  concept  in  relation  to  others  and  to  be  able  to  perceive  how  it  behaves  in  different  unfamiliar 
situations.  To  understand,  it  is  important  for  the  learner  to  make  connection  between  the  newly  acquired 
knowledge  about  the  concept  and  the  already  acquired  prior  knowledge  in  meaningful  ways  (Reigeluth,  1998). 
Through  these  connections,  the  learner  is  able  to  see  where  the  “new”  fits  into  “old”  ones,  consequently, 
understands  the  concept  meaningfully.  In  cognitive  theories,  this  process  is  defined  as  assimilating  new 
information  to  existing  knowledge  structures  and  accommodating  these  already  structured  knowledge  units  to 
the  new  information  as  necessary.  Thus,  it  is  important  to  have  agreeability  between  the  newly  coming 
information  and  existing  knowledge  unit  in  mind  is  important.  When  learners  get  exposed  to  information  units 
similar  to  those  already  structured  in  their  mind,  it  is  easier  to  activate  these  already  structured  units  (Winn, 
2004).  In  other  words,  showing  relevancy  of  new  information  seems  to  help  process  and  make  sense  of  the 
information  for  learners. 

When  teachers  were  asked  about  their  ways  to  help  students  recall  their  prior  knowledge,  the  main 
ways  they  des  cribed  were  (1)  reminding  students  the  prior  knowledge  as  necessary  before  or  during  teaching  a 
new  topic  verbally  or  on  the  board  and  (2)  having  students  memorize  information  from  prior  classes. 

John  and  Kurt  had  the  same  way  that  they  just  go  back  and  review  the  materials  when  they  think  that 
there  is  a weakness  in  students’  understanding  of  the  new  topic.  It  seems  that  time  spent  in  stimulating  the  prior 
knowledge  is  pretty  limited  especially  when  the  students  are  not  able  to  recall  the  “old”  knowledge  well  and 
there  are  many  new  topics  to  be  covered  in  one  lesson.  Kristin,  when  talking  about  recalling  prior  knowledge, 
continued  that: 

Within  the  range  of  the  students  we  have  in  one  class  that  some,  of  course,  recall  and  you  know  that’s 
the  hard  part. . . we  have  some  [students]  that  do  not  recall  at  all,  they  do  not  remember.  When  you  start  a new 
instruction,  [you  say]  “here  is  your  path  on  the  back  what  we  did  before,  you  remember  those”  and  through  that, 
“o  yeah  o yeah”  like  all  start  going  often  and  you  just  do  the  best  you  can  with  that  and  then  lead  them  slowly 
into  a new  topic. 

On  the  other  hand,  mostly,  students  are  expected  to  recall  what  they  have  already  learned.  George, 
explaining  his  way  of  stimulating  prior  knowledge,  mentioned  that: 

I have  them  memorize  quite  a bit,  they  have  to  know  the  statement  or  some  theorems.  I mean  I try  to 
tell,  “ you  need  to  know  the  exact  languages  or  you  need  to  be  able  to  talk  about  the  ideas  on  this  theorem”.  My 
point  to  them  is  “well,  it’s  just  like  learning  speaking  a language,  you  have  to  be  fluent,  and  you  have  to  know 
the  vocabulary”. 

Even  though  George  had  a good  point  in  knowing  certain  aspects  of  prior  knowledge  in  understanding 
new  ones,  he  did  not  appear  to  provide  certain  ways  of  helping  students  recalling  the  prior  knowledge  pool 
other  than  having  them  memorize. 

Kurt  noted  another  issue  in  recalling  prior  knowledge.  Even  though  he  said  that  he  reviewed  some  materials  as 
he  felt  that  students  needed  to  recall,  his  expectation  from  high-level  students  was  pretty  different  than  from  the 
low  ones: 

I teach  two  honor  classes.  I expect  them  to  know  certain  things  and  often  I am  disappointed  because 
they  do  not  recall  what  I hope  they  do. 

It  appeared  that  he  was  more  likely  to  review  materials  for  low  level  students  because  he  believes  that 
high  level  ones  can  remember  and  know  some  certain  aspects  already  but  in  fact,  often  “they  do  not  recall” 
either  as  he  mentioned  his  disappointment. 

Presenting  the  Stimulus  Learners  are  supposed  to  get  exposed  to  certain  types  of  information  in  order  to  make 
sense  of  it.  This  event  is  mostly  where  the  students  are  given  the  information  presentation  in  variety  of  ways 
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(Aronson,  1983).  Verbal  information,  demonstration,  visual  means,  and  others  are  the  general  ways  for 
presenting  a concept,  examples,  and  relevant  supportive  information. 

Drawing  on  the  board,  verbally  given  examples,  problems,  use  of  video  and  some  basic  computer 
applications  for  visual  representations  were  the  main  ways  which  the  teachers  mentioned  for  presenting 
stimulus.  However,  there  were  some  points  that  the  teachers  made  in  terms  of  how  students  could  have 
problems  with  a given  stimulus.  One  big  issue  that  George  was  concerned  with  is  the  complexity  and 
importance  of  language  level  in  math.  He  said: 

One  problem  that  they  have  1 am  convinced  is  just  the  language  level  in  math I mean  1 have 

students  that  can  read  a book  read  a page  in  an  English  class  and  it’s  a full  of  details  too  but  they  can 
bluff  it  by  not  having  to  read  that  quite  everything  in  English.  I mean,  in  math,  it’s  so  much  more 
compact  in  that  things  really  matter,  you  have  got  to  know  every  word  in  a sentence  and  what  it  means, 
there  is  a lot  of  language  issues. 

As  George  pointed  out,  the  first  matter  for  students  is  to  know  the  “every  word”  to  make  sense  of  the 
“sentence”.  Maybe  the  ways  in  which  information  is  presented  is  important  but  it  seems  that  it  may  not  matter 
for  students  to  see  the  information  in  different  forms  if  they  do  not  have  the  certain  prior  understanding  of  that 
information. 

Scared  of  Math?  When  talking  about  problems  that  students  can  face  in  math,  teachers  were  aware  that 
students  may  have  negative  feelings  about  math.  According  to  the  teachers,  unsuccessful  math  background  and 
lack  of  confidence  were  the  main  reason.  George  perceived  feeling  lack  of  confidence  and  feeling  hesitations  as 
a part  of  bad  interaction  between  students  and  teachers.  He  went  on: 

I think  a lot  of  teachers  are  open  to  questions . . . well,  I answer  any  kind  of  questions, ....  some  teachers 

would  talk  about,  you  know,  “students  are  asking  dumb  question  or  stupid  questions” if  you  asked  me  a 

question  in  math  what  3+4  is,  I would  tell  you  in  a strait  face.. ..  I want  them  to  really  feel  like  that  they  can  ask 
me  anything  in  math. 

He  had  the  same  remarks  when  suggesting  that: 

I know  that  again,  it’s  deadly  if  the  teacher  says  “ if  you  had  listened  to  me,  you  would  know  it,  you 

did  not  listen”.  I think,  we  have  to  avoid  this  [kind  of  remarks] I think  we  have  a lot  of  [this] 

kind  of  things  going  on  and  that  makes  students  reluctant  to  ask  because  they  do  not  wanna  ask  [at 
all]. 

In  addition  to  attributing  lack  of  confidence  to  the  bad  student-teacher  interaction,  George  also  viewed 
the  interaction  among  students  as  another  reason  for  a negative  attitude  about  math. 

....if  they  judge  the  smartest  person  in  class,  maybe  then  they  might  be  thinking  that  “I  am  the  dumbest  person 
in  the  class”  They  all  are  very  good  but  rank  themselves,  you  know,  for  good  or  bad.  I think  some  do  like 
competence. 

Examining  the  opinion  on  one’s  own  confidence  based  on  the  belief  about  other  people’s  ability, 
Wagner  (1984)  found  that  one’s  confidence  gets  higher  if  a superior  other  (in  ability)  is  perceived  to  agree  with 
him  whereas  the  confidence  becomes  lower  if  a superior  other  disagree  with  him.  In  other  words,  a student  can 
feel  less  confident  in  a classroom  setting  if  he  thinks  that  there  are  some  other  better  students  who  would 
disagree  with  him  when  he  talks.  Bandura  (1997)  notes  that  observing  or  perceiving  others  in  their  ability  to 
carry  on  a task  creates  a vicarious  experiences,  which  contributes  to  the  ones  own  beliefs  in  whether  he  can 
carry  the  same  task  as  well.  Likening  self  to  others  in  terms  of  ability,  accordingly,  can  be  helpful  for  one  to 
complete  a task  as  long  as  others  are  perceived  successful  in  completing  it.  On  the  other  hand,  if  one  lowers  his 
perception  about  his  own  skills  and  sees  those,  who  successfully  complete  tasks,  superior  to  him,  then,  the 
vicarious  experience  would  not  benefit  him.  In  fact,  it  would  create  negative  feelings  and  lead  to  lowering 
expectations  about  his  performance  on  a given  task. 

Kristin,  on  the  other  hand,  made  another  point  in  explaining  why  students  may  have  negative  attitudes 
toward  math.  She  believed  that  parents’  thoughts  and  experiences  on  math  affect  students’  confidence  about 
math.  Based  on  her  conversation  with  parents,  she  commented  that: 

At  home,  they  [-students]  are  told  that  “ I can  not  help  you  with  this  homework  I do  not  understand 
math”  and  parents  tell  them  right  a way  that  it  is  ok  not  to  understand  it  and  you  know  “I  am  afraid  to  try  to  look 
at  it  with  you”  so  I think  they  are  discouraged,  some  of  very  early  age. 

This  might  be  the  same  thing  as  verbal  persuasion  that  Bandura  describes  (1997)in  terms  of  how  one’s 
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belief  about  his  ability  to  carry  on  a task  can  be  affected  by  others’  ideas,  encouragement,  or  discouragement 
about  the  person’s  potential  performance.  In  this  case,  parents’  beliefs  about  their  own  low  ability  in  math  and 
consequent  low  expectations  of  their  own  kids’  ability  might  serve  as  a discouraging  factor  for  kids  to  develop 
confidence  in  math. 

Kristen  also  talked  about  family  meetings  in  which  teachers  introduce  short  class  sessions  to  parents. 
She  sees  this  as  a way  of  changing  parents’  perceptions  about  math,  consequently,  helping  students  not  have 
negative  attitudes  toward  math: 

This  is  our  third  year,  in  the  first  year  we  had  30  parents  and  students  come,  last  year  it’s  about  120, 
this  year  300.  So,  what  they  do  is  that  we  have  teachers’  teaching  25min.  session  using  graph  and  calculator, 
there  are  simple  things,  using  GEO  boards,  origami.  Parents  are  terrified  to  come,  you  see  them  when  they  walk 
in  the  door,  we  have  pizzas,  we  talk  to  them  a little  bit,  in  the  end,  they  all  just  smile. 

If  parents’  perceptions  are  really  considered  to  be  influencing  their  kids  perceptions  about  math,  these 
meetings  really  seem  to  help  parents  and  students  respectively. 

The  Program’s  Contribution  to  the  Teaching  Practices 

After  talking  about  their  teaching  practices,  the  teachers  were  asked  whether  being  in  the  program 
contributes  to  their  teaching  practices.  It  seems  from  the  teachers’  comments  that  the  program  influenced 
teachers’  teaching  practices  in  a broader  but  not  direct  way.  They  see  the  program  as  an  opportunity  to 
collaborate  with  other  teachers,  do  lesson  plans  elaborately,  do  pedagogical  readings  for  their  own  instruction. 
When  talking  how  the  program  affects  his  way  of  teaching,  Kurt  emphasized  that : 

I think  the  program  has  not  necessarily  influenced  how  I assess,  how  I present,  or  how  I communicate 
[with  students]  but  it  is  all  in  there  together.  It  makes  me  concentrate  on  certain  aspects  of  my  own  instruction. 

For  one  of  these  “certain  aspects  ” , he  added  that: 

I am  in  the  multiple  representation  group  and  I have  found  myself  trying  to  include  multiple  representations  of 
algorithmic  relationships.  I think,  it’s  important  for  students  to  see  relationships  in  multiple  ways  because  they 
are  not  gonna  understand  [the  relationships  in]  one  way. 

Kristin  also  noted  how  her  teaching  practices  became  different  as  a result  of  joining  the  program: 
....there  are  three  multiple  representations  that  I used  and  I really  think  it  helped.  By  the  end,  the  kids  are  saying 
“ oh  I know  how  to  do  it  in  algebraically,  can  I just  write  it  all  out”....  Before  I have  not  thought  about  doing 
these  three  things,  I was  just  struggling  to  get  them  to  write  it  out  in  one  of  the  ways.  You  know,  they  would  just 
make  statements  like  they  would  put -2  and -2  on  each  site  rather  than  writing  a new  statement  that  is  actually 
new  something  mathematically.  So  they  are  begging  me,  “I  am  ok  with  this,  do  I have  to  draw  it  anymore?,  I 
just  wanna  write  it  all  out”,  and  then,  I am  like  “ you  just  want  to  write  it  out,  ok,  that’s  fine”. 

Based  on  similar  comments,  it  is  pretty  obvious  that  the  program  affected  teaching  practices  of  the 
teachers  by  not  only  enabling  teachers  to  focus  on  specific  elaborated  lesson  plans  but  also  encouraging  teachers 
to  promote  different  multiple  ways  for  students  to  understand  and  learn.  The  teachers  also  agree  that  the 
program  contributes  to  their  professional  development  by  giving  opportunities  such  as  going  to  conferences, 
using  technologies  and  resources  in  their  classroom  etc.  On  the  other  hand,  there  are  some  suggestions  that  the 
teachers  made  how  meetings  could  be  enhanced.  More  reflective  and  interactive  discussions  are  what  the 
teachers  would  like  to  have.  Regarding  how  group  meetings  should  be  taking  place,  George  noted  that: 

I think  I like  that  we  talk  to  the  whole  group  a little  bit  about  what  our  small  groups  are  doing  because  I 
feel  like  I want  to  share  with  everybody  what  we  were  talking  about  in  our  proof  [group]  like  “ we  are  trying  to 
get  this  theme  of  justifying  more  often  in  class”....  In  proof  [group]  we  have  got  something  to  start  to  share  and 
the  multiple  representation  group  talked  about  specific  lesson  plan  but  when  we  ask  that  they  said  something, 
they  did  talk  about  what  kind  of  their  overall  goal....  I enjoyed  doing  that,  but  I like  to  have  some  chances  to 
hear  some  more  things  that  may  have  impact  on  all  the  teachers. 

It  seems  like  the  subgroups  of  the  program,  even  though  they  are  active  within  their  own  groups,  need 
to  “be  heard”  and  need  to  “hear”  in  other  subgroups  as  well.  More  feedback  from  other  groups  is  something 
Kurt  would  like  to  have: 

I would  like  to  get  more  people  speak  on  what  we  are  doing  in  our  sub  groups I wish  that  there  were 

ways  to  get  people  to  give  me  more  feedback  on  my  teaching  practices the  more  feedback  I can  get  on 

what  I do  in  my  classroom,  the  better  teacher  that  would  make  me. 
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Enabling  teachers  to  communicate  with  each  other  more  and  provide  with  feedback  necessarily  seems 
to  be  a good  point  that  the  program  needs  to  take  into  account.  Leading  the  teachers  to  use  communication  tools 
such  as  online  discussions  could  be  one  example  as  George  mentioned  with  regard  to  an  existing  discussion  tool 
[ILF]: 

There  have  been  times  that  [ILF]  was  used  effectively  but  for  whatever  the  reason,  we  have  not  used  ILF 
resources  in  terms  of  the  facility  to  work  on  a common  document  or  the  chat.  I thought  it  would  be  worth  trying 
to  some  time  to  have  discussion  online  but  we  have  never  done  that. 

Conclusion 

In  this  paper,  four  high  school  mathematic  teachers’  teaching  practices  and  their  professional 
development  experiences’  contribution  to  these  practices  was  documented.  Not  only  the  professional 
development  program  was  expected  to  benefit  from  this  documentation  but  also  many  educators,  such  as 
instructional  design  and  theory  developers  were  assumed  to  learn  from  these  teachers  teaching  experiences. 

For  the  Program 

It  is  obviously  seen  that,  the  teachers’  initial  expectations  are  pretty  met  with  the  support  from  the 
program.  Collaborating  with  other  teachers  in  meetings  to  discuss  lesson  plans  in  details,  doing  pedagogical 
readings  for  their  own  instruction,  providing  feedback  to  each  other,  being  reflective  about  the  teaching 
practices,  and  being  in  a professional  development  are  those  the  teachers  expected  to  have  and  mostly  were 
provided  by  the  program.  There  are  many  ways  that  the  teachers  incorporate  their  experiences  from  the  program 
to  their  teaching  practices.  Presenting  students  in  multiple  ways  that  were  mentioned  in  the  group  meetings  is 
one  of  the  obvious  direct  pedagogical  contributions  of  the  program.  On  the  other  hand,  there  could  be  some 
ways  to  improve  the  program  as  a better  professional  development.  (1)  Having  more  and  easy  discussion 
environments  among  the  teachers  such  as  those  online  communications  and  (2)  providing  more  constructive 
feedbacks  not  only  within  subgroups  but  also  among  the  groups  are  the  main  ones  noticed  by  the  teachers. 

For  Educators 

Regarding  the  teaching  practices,  it  seems  that  there  are  many  similar  and  different  ways  that  teacher 
applies  in  each  nine  events  of  instruction.  In  addition  to  that,  there  some  issues  that  the  teachers  pointed  out  for 
implementation  in  some  of  the  events.  Below  is  a general  table  that  describes  the  strategies  applied  and 
important  issues  considered  by  the  teachers  in  these  nine  events. 


Table  1-  Teachers'  Ways  in  Each  9 Events  and  Possible  Corresponding  Issues 


Events 

Teachers’  ways  in  events 

Issues  mentioned 

Gaining 

attention 

> Questioning 

> Announcements 

> Starting  with  Problems 

> Starting  with  High  level  of  Tasks 

> Listing  Objectives 

> Yelling  etc. 

> Keeping  students  busy  and  focused  on  certain  tasks 

> Putting  students  in  pairs  and  letting  them  discuss 

Students  can  get  frustrated 
when  confronted  with  a 
difficult  questions  or  high 
level  of  tasks. 

Informing 
learners  of 
the  objective 

> Announcing  verbally 

> Written  objectives  on  the  board 

> Written  objectives  in  handouts 

It  may  take  too  long  so  there 
may  be  not  enough  time  to 
start  and  go  on  the  new  topic. 

Stimulating 
recall  of 
prior 
learning 

> Reviewing  necessary  prior  information 

> Verbally  reminding  students  the  prior  knowledge  during 
or  before  the  presentations  of  a new  topic 

> Having  students  memorize  prior  knowledge  and  expecting 
them  to  recall 

> Providing  resources  to  students  as  much  as  they  need 

Students  have  to  have 
necessary  prior  knowledge  but 
it  might  be  hard  for  them  to 
recall  everything. 
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Presenting 
the  stimulus 

> Using  Visual  tools  (powerpoint,  graphics,  drawing  on  the 
board,  graph  calculator,  etc.) 

> Using  Numerical  Representations 

> Giving  students  enough  time  and  Letting  them  discover 

> Giving  examples  and  problems 

Students  may  have  different 
preferences  regarding  the 
teaching  way  -structured  Vs 
less  structured  lesson 

Students  may  have  problems 
when  being  exposed  to 
stimulus  and  trying  to  make 
sense  of  it  at  the  same  time 

Providing 

learning 

guidance 

> Providing  handouts, 

> Giving  more  examples 

> Re -explaining  points  made  in  books 

Not  mentioned 

Eliciting 

performance 

(Practice) 

> Problem  solving  after  each  concept  or  before  the  class  is 
over 

> Encouraging  students  to  solve  on  the  board 

Not  mentioned 

Providing 

feedback 

> Showing  students  their  own  mistakes  rather  than  showing 
the  all  solutions. 

> Using  positive  reinforcements  (e.g  Me  Donald  cards  ) 

> Asking  students  to  provide  each  other  feedback  in  group 
work 

Not  mentioned 

Assessing 

performance 

> Questioning  students  for  their  response 

> Controlling  homework  and  journals  completions 

> Doing  Quizzes,  tests,  etc. 

Not  mentioned 

Enhancing 
retention  and 
transfer 

> Putting  things  in  tests  from  prior  class  sessions 

> Asking  students  to  take  notes, 

> Making  connection  to  previously  acquired  knowledge 
through  reviewing  and  reminding 

> Giving  Homework 

Not  mentioned 

Among  these  nine  events,  gaining  students  attention,  stimulating  recall  of  prior  learning,  presenting  the 
stimulus  are  the  ones  that  the  teacher  seemed  to  focus  on  more.  Teachers  also  have  different  types  of  strategies 
and  problems  that  they  can  encounter  in  each  event.  According  to  the  teachers,  challenging  tasks  as  a way  of 
gathering  students’  attention  could  be  problematic  depending  on  the  level  of  the  students.  Also  the  lack  of 
resources  to  use  in  solving  these  challenging  problems  could  be  frustrating  for  students.  On  the  other  hand, 
regarding  the  stimulating  the  prior  learning,  students  may  be  expected  to  recall  instead  of  being  help  to 
remember.  This  sounds  problematic  if  they  are  only  expected  to  memorize  and  be  “fluent”  because  like  in  many 
subject  matters,  understanding  concepts  heavily  depends  on  the  connections  that  the  student  can  make  between 
what  he  is  learning  and  what  he  has  already  known.  Presented  the  stimulus,  students  may  have  problems  in  both 
trying  to  take  notes  (being  exposed  to  the  stimulus)  and  trying  to  understand  what  the  stimulus  means.  Even 
though  they  have  a chance  to  review  the  notes  that  they  take  later,  it  may  be  hard  for  them  to  understand 
something  that  they  did  not  in  class.  Another  issue  in  this  event  is  the  preferences  that  students  may  have  for 
their  learning  styles.  For  those  who  “resist”  discovery  approach,  using  more  structured  information  presentation 
may  seem  to  the  teachers  as  a good  strategy  but  in  this  case,  those  who  are  willing  to  “discover”  are  not  given 
the  opportunity  to  discover.  The  ideal  way,  as  emphasized  in  the  interviews,  could  be  a combined  approach  in 
which  students  are  exposed  to  both  discovery  and  more  structured  ways  of  learning  so  that  everyone,  at  least, 
can  have  a chance  (to  some  degree)  to  learn  in  way  that  they  prefer. 

The  teachers  also  mentioned  the  reasons  of  negative  feelings  that  students  might  have  toward  math. 
Lack  of  confidence  due  to  lack  of  positive  feedback  from  teachers,  seeing  other  students  superior  to  self, 
discouraging  prompts  from  the  teacher,  and  low  expectations  that  parents  have  about  their  kids’  performance 
were  perceived  as  main  reasons.  More  caring  and  positive  communications  between  teachers  and  students  and 
more  encouragement  from  parents  about  the  students’  performance  are  the  ways  that  the  teachers  perceived  to 
handle  the  students’  negative  attributions  to  math. 

As  George  mentioned,  “ it  is  tough  to  have  the  appropriate  thing  for  every  student ” but  it  is  always 
worth  trying  to  have  multiple  ways  and  necessary  communications  to  approach  the  students  in  a caring, 
motivating  but  not  discouraging  ways  so  that  their  learning  can  be  enhanced  by  letting  them  go  beyond  their 
possible  fears  and  learning  obstacles. 
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Forming  Virtual  Learning  Community  within  Online  Course:  Students’ 

Perspectives 

Fengfeng  Ke 
Penn  State  University 

Abstract 

This  research  intends  to  explore  students’  aspect  and  impact  oil  forming  virtual  learning  communities 
within  online  courses.  A formal  online  course  that  is  on  the  process  of  building  virtual  community  is  closely 
examined  to  discover  students  ’ perceptions  of  learning  community  approach,  their  online  performances,  and 
how  these  perceptions  and  performances  facilitate  or  hinder  the  forming  of  a virtual  learning  community.  The 
study  can  serve  as  a useful  guide  for  online  education  practitioners  and  online  learners. 

Introduction 

In  this  information  era,  institutions  of  higher  education  are  delving  into  the  world  of  online  learning: 
there  has  been  rapid  growth  in  the  number  of  course  being  offered  either  entirely  online  or  as  a supplement  to  a 
face-to-face  course  (Underwood  et  ah,  2000). 

On  the  other  hand,  online  courses  have  been  appearing  so  rapidly  that  little  thought  or  effort  seems  to 
be  given  to  the  specific  needs  of  the  classroom  in  cyberspace.  Traditional  teaching  methods  are  being  attempted 
in  a nontraditional  environment  (Palloff  & Pratt,  1999).  Many  online  courses  simply  put  face-to-face  class’s 
lecture -based  content  online.  Such  kind  of  “traditional”  online  courses,  as  demonstrated  by  the  research 
(Besser,  1996;  Carr,  2000;  Herrington,  et  ah,  2001,  Kerka,  1996;  Schrum,  1995,  Swan,  2001),  have  caused  high 
dropout  rates,  bad  learning  outcome,  and  low  satisfaction  rate.  Therefore,  how  can  we  modify  current 
educational  strategy  to  enhance  online  learning  process?  In  answering  the  question,  educational  researchers 
have  recurrently  proposed  the  virtual  learning  community  approach. 

Briefly,  virtual  learning  community  is  an  extension  of  the  physical  learning  community  outward  to  the 
electronic  one  (Russell  & Ginsburg,  1999),  which  is  originated  from  the  constructivism  learning  theory  that 
proposes  a goal-based  collaborative  learning  within  a community  context  (Wenger,  1998).  Palloff  and  Pratt 
(1999)  have  explained  the  importance  of  community  in  electronic  classroom  by  arguing,  “The  learning 
community  is  the  vehicle  through  which  learning  occurs  online... Without  the  support  and  participation  of  a 
learning  community,  there  is  no  online  course.”  The  importance  of  community  in  online  learning  is  also 
supported  by  empirical  research.  Quite  a few  studies  (e.g.  Hiltz,  1998;  Prestera  & Moller,  2001;  Russell,  1999; 
Russell  & Ginsburg,  1999;  Shrivastava,  1999;  Wang,  et  ah,  2001)  evidence  that  virtual  learning  community  is  a 
powerful  tool  to  boost  online  learning  participation  and  achievement. 

A virtual  learning  community,  according  to  Rovai  (2002),  can  be  constitutively  defined  in  terms  of 
four  dimensions:  spirit,  trust,  interaction,  and  commonality  of  goals  (learning).  Similar  definitions  of  virtual 
learning  community  can  be  also  seen  in  the  works  of  McMillan  & Chavis  (1986),  Jonassen,  Peck,  & Welson 
(1998),  and  Kowch  & Schwier  (1997).  These  definitions  suggest  an  essential  framework  of  a virtual  learning 
community,  which  comprises:  an  active  learning  environment  that  fosters  a climate  of  learning  in  community,  a 
dynamic  learner-directed  process  of  “communication,  collaboration,  interaction,  and  participation”  (Lock,  2002, 
p.  397),  and  the  development  of  feeling  or  sense  of  community. 

However,  even  though  literature  has  explained  why  virtual  learning  community  is  important  and  what 
it  is,  another  fundamental  question  remains  poorly  answered  - how  to  build  a virtual  learning  community? 

Currently,  there  are  a select  few  studies  examining  the  development  of  learning  communities  within 
online  courses.  These  studies  focus  on  the  community  environment  and  address  the  issue  through  the  lens  of 
designers  and  developers.  Some  attempt  to  describe  a systematic  development  model  for  building  virtual 
learning  community,  including  the  research  by  Barker  (2001),  Lock  (2002),  and  Ravitz,  (1997).  For  instance, 
Ravitz  (1997)  proposed  a seven-stage  ISD  model  for  building  virtual  communities,  which  involves 
management,  front-end  analysis,  communication  environment  design,  projects  development,  implementation, 
community  evaluation,  and  information  dissemination. 

Others  (e.g.  Kuhl,  2002,  Nixon  & Leftwich,  2002;  Prestera  & Moller,  2001;  Quitadamo  & Brown, 

2001;  Snyder,  2002;  Yoder,  2003)  emphasize  the  pedagogical  issues  when  designing  instructional  context  that 
promote  community.  A representative  one  is  done  by  Yoder  (2003)  who  presents  seven  strategies  to  foster  a 
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community -promoting  online  communication  process. 

Finally,  some  design  research  (e.g.  Cutbbert,  et  al.,  2002;  Jin,  et  al.,  2001;  Lally  & Barrett,  1999)  has 
evolved  around  the  technological  issues  in  constructing  a virtual  “community  place”.  For  example,  Lally  and 
Barrett  (1999)  discussed  the  strategy  of  using  computer- mediated  communication  to  reduce  transactional 
distance  and  facilitate  the  construction  of  learning  communities  in  an  online  environment. 

Few  of  these  studies,  however,  are  empirical  research.  Additionally,  previous  literature  has  not  amply  described 
students’  perspectives  and  impact  in  community  development.  As  Lock  (2002)  and  Jonassen,  et  al.  (1998)  have 
pointed  out,  the  learning  community  model  depends  largely  on  students.  Students  need  to  be  aware  of  the 
community  philosophy  and  “make  a paradigm  shift  in  their  learning  strategies”  (O’Sullivan  & Miron,  2000,  p. 

7).  However,  guidelines  on  how  students  respond  to  community  in  cyberspace  are  scarce. 

As  a conclusion,  significant  work,  especially  empirical  research,  is  needed  to  well  address  the  issue  of  building 
community  within  online  course  in  terms  of  learners’  perspectives,  responses,  and  impact  on  community 
forming. 


Research  Purpose  and  Questions 

This  qualitative  study  intends  to  explore  learners’  perspectives  and  responses  to  forming  virtual 
learning  communities  within  online  courses.  Specifically,  the  central  research  question  to  be  answered  is:  what 
are  students’  responses  to  the  virtual  learning  community  and  how  do  these  responses  relate  to  the  community 
development? 

This  research  contributes  to  the  literature  by  attempting  to  shed  light  on  the  development  process  of 
virtual  learning  community  in  terms  of  what  is  really  happening  to  participants  and  how  their  responses 
influence  the  forming  of  the  community. 


Methodology 

The  research  is  designed  as  a qualitative  case  study.  According  to  Stake  (1995)  and  Creswell  (1998), 
case  study  is  employed  when  the  project  involves  a “case”  (bounded  in  time  or  place)  and  extensive  materials 
from  multiple  sources  are  collected  to  provide  an  in-depth  picture  of  the  “case”. 

In  this  study,  a formal  online  course  in  a major  American  university,  which  takes  learning  community 
approach  in  instructional  context  design,  is  identified  as  a “bounded  case,”  bounded  by  time  (5  months  data 
collection)  and  place  (a  single  course).  Extensive  data  from  individual  in-depth  interviews,  online  activities 
observation,  email  messages,  transcripts  of  bulletin  board  and  chat  room  discussions,  students’  assignments,  and 
other  course  materials  have  been  collected.  Through  this  data  collection,  the  researcher  then  makes  a detailed 
description  of  the  context  or  settings  of  the  case,  conducts  an  analysis  of  themes,  and  gives  an  interpretation  or 
“assertions”  about  the  case  (Stake,  1995). 

The  Case  Setting 

INSYS  446  (spring,  2004)  was  a purely  online  graduate  course  delivered  by  a major  American 
university  through  ANGEL  course  management  system.  The  course  was  mediumsized  (with  30  students).  The 
instructor  of  the  course  has  taken  a constructivism  learning  community  initiative  to  design  and  develop  the 
instructional  context. 

First,  instructional  and  learning  activities  evolve  around  intensive  online  interactions  and 
collaborations  using  both  synchronous  (chat  room)  and  asynchronous  (bulletin  board)  conferencing  tools. 

Online  interaction  and  discussion  is  required  and  graded.  The  syllabus  lays  out  a clear  specification  on  the 
frequency  and  content  of  the  peer  feedback.  Group  work  with  project-oriented  collaboration  and  information 
exchange  is  emphasized  in  the  course  assignment.  Students  need  to  develop  application  projects  in  the  unit  of 
small  group.  This  fact  is  explicit  in  the  following  description  cited  from  the  course  syllabus: 

You  will  work  in  teams  of  3 when  possible  and  each  team  will  create  three  mindtool  projects.  We  will 
begin  with  a planning  week  then  have  three  production  weeks.  You  will  act  as  team  leader  for  one  of  the 
production  weeks  and  team  member  for  the  other  two.  Each  team  leader  will  guide  the  production  of  a mindtool 
using  a different  software  package.  Team  leaders  can  ask  teammates  for  assistance  with  development  and 
assessment  ideas,  and  will  ask  teammates  to  test  the  mindtool.  Each  class  member  will  act  as  a codeveloper  on 
2 mindtools  projects.  Codevelopers  will  advise  team  leaders  and  serve  as  beta  testers  of  mindtool  projects 
(INSYS  446,  March  8th,  2004) 

Second,  the  learning  process  is  student-directed  rather  than  lecture -based.  In  the  introduction  part  of 
the  course  syllabus,  the  instructor  claims  that  he  will  let  students  explore  the  subject  before  sharing  his  opinions 
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with  them.  This  claim  can  be  observed  from  the  fact  that  on  the  course  site  there  is  no  section  for  explicit 
lectures.  Online  instruction  is  conducted  in  forms  of  feedback  and  guidance,  through  bulletin  board  and  emails. 
Learning  is  active  and  project-based:  Students  need  to  develop  projects  from  real  world  problems  to  illustrate 
their  knowledge  of  subject  content;  they  share  expertise  and  contribute  multiple  perspectives  to  peers  during 
project  development  and  assessment;  in  addition,  they  are  provided  opportunities  of  taking  on  various  roles 
(leadership  or  regular  member)  in  support  of  their  learning  process.  Such  a project-based  situated  learning,  as 
Tam  (2000)  and  Lock  (2002)  suggest,  helps  to  cultivate  an  environment  that  promote  community. 

Third,  the  instructor  encourages  an  “atmosphere  of  adventure”  (Hill,  2001,  p.  9)  in  his  evaluation 
mechanism.  Learning  tasks,  whether  reading  review  or  application  projects,  are  mo  stly  open,  heuristic,  not 
assuming  only  one  standard  answer.  Students’  assignments,  as  observed,  displayed  multiple  presentation 
format  and  multiple  perspectives.  They  are  evaluated  more  in  terms  of  efforts  and  richness,  rather  than  absolute 
correctness. 

Finally,  the  setting  of  the  online  discussion  has  emphasized  group  cohesiveness  and  identity.  In  the 
bulletin  board,  each  group  has  its  own  group  discussion  forum  in  addition  to  the  general  class  forum,  and  each 
group  has  a specific  name  for  identification,  such  as  “Banana”  team,  “Apple”  team,  and  the  sort.  Then,  the 
grouping  is  developed  based  on  two  criteria:  interest  in  the  same  learning  topic  choice,  or  the  same  professional 
background  that  rear  interest  in  similar  real  world  problem.  This  kind  of  grouping  strategy,  as  the  researcher 
interpreted,  is  a reflection  of  “commonality  of  expectation”  among  group  members  (Rovai,  2002). 

During  observation  and  interviewing,  the  researcher  has  not  found  an  obvious  concern  on  relationship 
development  in  INSYS  446’ s instructional  design  and  development.  However,  the  instructor  does  allocate  one 
orientation  week  for  students  to  post  a detailed  self-introduction  message  to  the  whole  class.  He  also  opens  a 
personal  page  for  each  student  and  asks  them  to  upload  pictures  and  personal  information  there.  Most  of 
participants  interviewed  express  their  appreciation  of  the  orientation  and  personal  pages  as  customs  to  develop 
sense  of  familiarity  and  relationship. 

Based  on  the  above-mentioned  features,  the  researcher  believes  that  INSYS  446  (spring,  2004)  is  a 
representative  and  valuable  case  to  be  examined  for  the  research  intended.  Hence  the  researcher  asks  for  the 
instructor  for  the  permission  to  access,  and  has  examined  the  case  as  an  avid  spectator. 

Participants 

Creswell  noted,  “The  purposeful  selection  of  participants  represents  a key  decision  point  in  a 
qualitative  study”  (1998,  p.  118).  By  following  his  “maximum  variation”  strategy  to  select  subjects  that 
represent  diverse  perspectives,  the  researcher  has  selected  12  participants  from  the  students  enrolled  in  INSYS 
446  (spring,  2004),  who  are  diverse  in  terms  of  age,  gender,  nationality,  professional  background,  prior  online 
learning  experience,  and  finally,  learning  styles. 

These  14  participants,  aged  from  20s  to  50s,  5 females  and  9 males,  comprise  full-time  education- 
majored  graduate  students  and  part-time  adult  students  who  are  pursuing  Instructional  Design  certificates.  Two 
of  them  are  non-American,  one  Korean  and  one  Chinese.  The  participants  come  from  different  career  fields:  K- 
12  education,  corporate  training,  organization  consultant,  and  higher  education.  They  also  vary  in  their  prior 
online  learning  experiences:  some  of  them  are  very  expert  online  learners  (having  taken  2 to  3 e-courses  before) 
while  others  have  INSYS  446  as  their  first  online  course.  At  the  beginning  of  the  study,  all  participants  have 
taken  the  Myers -Briggs  Type  Indicator  (MBTI)  (Association  for  Psychological  Type,  2000)  and  the  Cognitive 
Styles  Assessment  (CSA,  Riding  & Rayner,  1998).  Their  test  results  indicate  multiple  personality  types  along 
the  four  personality  dimensions  (extraversion/introversion,  sensing/intuition,  thinking/feeling,  and 
judging/perceiving)  and  different  cognitive  styles  (the  ratio  of  field  dependent  to  field  independent  being  6 to 
8). 

Data  Collection 

To  enhance  the  vigor  of  the  study,  the  investigators  have  employed  data  triangulation,  which  includes 
interview,  observation,  and  discourse  analysis.  The  interview  was  individual,  semi -structured,  extended,  and 
iterative.  Concurrently,  the  investigators  have  observed  these  subjects’  online  discussion  activities  by  both 
reading  through  their  on-going  message  exchange  and  personally  observing  them  in  their  home  space.  Finally, 
the  investigators  have  als  o reviewed  course  documents,  class  emails,  and  bulletin  board  scripts  (of  10  weeks)  to 
examine  subjects’  online  participation  process. 

In-depth  interview:  The  researcher  conducted  individual  in-depth  interview  with  every  participant, 
each  interview  lasting  for  one  hour.  The  interview  was  semi  -structured:  an  interview  protocol  with  open-ended 
questions  was  framed  to  activate  the  exploring  of  interviewees’  stories.  The  researcher  did  face-to-face 
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interviews  with  two  participants  who  lived  around  the  school  area  and  telephone  interviews  with  the  others.  All 
interviews  were  tape-recorded.  The  participants  were  encouraged  to  explore  their  responses  to  virtual  learning 
community  to  the  fullest. 

Observation'.  The  observation  in  this  study  took  two  forms:  online  and  face-to-face.  For  the  online 
observation,  the  researcher  logged  into  the  course  site  and  read  through  participants’  postings  in  bulletin  board 
and  chat  room.  A semi  -structured  observation  protocol  was  developed  to  guide  the  attention  during 
observation,  though  the  actual  observation  was  open  to  any  situational  changes.  The  researcher  also  personally 
observed  two  participants  when  they  logged  onto  course  site  and  did  online  posting  at  their  study  rooms. 

Document  analysis'.  Course  documents,  such  as  the  syllabus,  course  timeline,  students’  projects,  and 
their  class  emails,  were  also  collected  and  coded. 

Data  Analysis 

By  following  Stake’s  (1995)  proposition  on  the  case  study,  the  researcher  has  first  aggregated  the  data 
into  about  24  categories  (categorical  aggregation)  and  collapsed  them  into  two  patterns  with  nine  themes. 
Member  checking  has  been  employed  to  ensure  the  credit  of  direct  interpretation.  Finally,  generalizations  about 
the  case  in  terms  of  patterns  and  themes  are  developed  in  comparison  with  the  published  literature  on  virtual 
community  development. 


Findings 

Two  general  patterns  with  nine  themes  have  emerged  through  the  process  of  data  generation,  analysis, 
and  comparison  of  participants.  These  central  patterns  and  themes  are  listed  and  discussed  below.  Participants 
have  been  referred  as  “PI”  or  “P2”  in  the  quotations. 

Cognitive  Response 

The  cognitive  response  involves  participants’  learning  strategies  and  actions  to  the  collaborative 
learning  in  a community  context . These  strategies  include  meaningful  communication,  interacting 
academically,  managing  group  work,  instructor-monitored  participation,  and  self -adaptation. 

Meaningful  communication : The  participants  expressed,  “typically  I would  not  just  post,  you  know,  for 
the  sake  of  it.”  In  other  words,  meaningless  communication  is  deemed  non-necessary.  They  cared  a lot  about 
the  content  of  discourse.  It  was  observed  that  the  participants  did  not  respond  to  all  the  postings.  Actually,  they 
purposely  selected  message  to  reply,  based  on  their  evaluation  on  whether  the  discourse  was  constructive  or  not. 
PI  said: 

When  I respond,  I respond  to  the  ones  that  I feel  that  I have  a comment  that  is  worthy  of  making 
posting.  I don’t  necessarily  always  want  to  put  down  “yes  I agree”  “no,  I don’t”.  I want  to  have 
something  that  I can  add  to  the  conversation.  So  I read  a lot  more  postings  than  I respond  to.  I don’t 
respond  to  all  the  postings.  I don’t  think  that  is  necessary. 

The  messages  that  I tended  to  respond  first  are  ones  that  are  practically  based,  usable  in  the  real  world. 
At  the  same  time,  the  participants  would  like  to  post  meaningful  messages  to  peers.  By  “meaningful”, 
participants  meant:  being  able  to  inform,  or  being  able  to  help.  For  instance,  participants  explained  the  usual 
occasions  when  they  would  post  messages: 

Let’s  say  somebody  asked  questions  that  I thought  I had  answer  for  or  suggestions  I might  post  that. 

Or  maybe  somebody  made  a comment  that  I could  maybe  add  additional  information  to  or  my  own 
personal  opinion  that  were  my  expand  on  that  comment,  I might  add  that  (P2). 

Usually  when  someone  who  are  new,  need  help  with  something,  I would  like  to  help  and  tell  (P8). 

In  order  to  ensure  the  “meaningfulness”  or  quality  of  their  discourse,  some  participants  stated  that  it  was  helpful 
for  them  to  do  a careful  pondering  before  posting  any  messages  : 

And  that’s  why  I liked  the  (discussion)  board:  it  is  because  it  gives  you  opportunity  to  think  about  what 
you  are  going  to  say  just  before  you  blur  it  out.  Sometimes  I will  start  to  write  a posting  and  then 
decide  that  “well,  I don’t  want  to  say  that”  so  I will  delete  it,  I won’t  post  it  (P3). 

It  is  hard  to  get  across  which  you  are  trying  to  say  without  careful  thinking  or  something,  so  I will  try 
to  reread  what  I typed  in,  I will  rearrange  sentences,  it ‘s  more  like  typing  a memo.  And  then  I will 
send  it.  Usually  I try  to  be  conscious  on  what  I am  posting  (P10). 

I prefer  online  discussion  to  face-to-face  one.  With  online,  you  can  have  a time  to  think  it  over  before 
responding  to  the  other  persons  (P4). 
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With  all  these  messages,  the  participants  demonstrated  that  they  were  concerned  about  the  content  or  the 
knowledge  density  of  a discourse.  They  believed  that  a worthy  discourse  was  filled  with  “valid  and  important 
information’'  rather  than  “gossiping"  (P3).  They  even  used  “being  usable  in  the  real  world"  as  a criterion  to 
judge  the  value  of  a message.  Therefore,  they  felt  disappointed  with  a simplistic  exchange  of  “yes,  I agree"  or 
“good  job”  which  supported  none  of  “information  exchange"  but  only  “social  reinforcement”  (Moller,  1998). 

Interacting  academically  more  than  socially:  The  interactions  among  the  participants,  as  they  put, 
“have  been  on  academic  level”.  In  coding  the  participants’  online  discourses,  the  re  searcher  found  that  almost 
all  the  messages  they  posted  or  responded  were  around  the  projects  to  be  developed  or  class  readings.  Few  of 
postings  were  to  fulfill  a social  function,  such  as  the  exchange  of  personal  information  or  feelings.  This  finding 
was  also  confirmed  by  the  interviewing  data.  For  instance,  some  participants  said: 

I don’t  know.  I see  this  more  as..  .1  guess  my  interactions  have  been  on  professional  level.  I haven’t 
been  contacted  on  personal  level,  nor  have  I contacted  anyone  on  personal  level.  There  is  not  enough 
time  for  that.  Maybe  because  people  are  just  so  spread  out  that,  that  has  an  effect  on  how  much  you 
want  to  invest  in  developing  an  relationship  that  would  probably  just  last  over  a couple,  you  know, 
several  weeks  (P5). 

I think  we  were  in  touch  academically,  not  necessary  socially.  We  had  little  social  talk.  What  is  the 
point?  I probably  will  never  meet  these  people.  Why  bother  some  social  relationship?  (P6) 

These  explanations  indicated  that  timing  and  physical  distance  were  two  practical  concerns  that  deterred  the 
social  interaction.  However,  beyond  these  surface  reasons,  a hidden  explanation  is  that  the  participants  did  not 
value  social  reactions.  They  wondered  about  the  meaning  of  social  talk  (by  saying  “what  is  the  point”)  and 
deemed  the  investment  in  developing  social  relationship  as  extra  or  non-necessary  (by  questioning  “why 
bother”). 

The  social-networking-relative  interactions  happened  mostly  during  the  first  orientation  week  when  the 
instructor  required  students  to  introduce  themselves  to  each  other  and  each  person  published  their  personal 
pages.  A point  to  note  is  all  participants  interviewed  expressed  their  appreciation  of  the  orientation  week  and 
peers’  personal  pages.  As  P3  told,  “At  least  now  I know  who  I will  work  with  in  the  team  and  where  he  comes 
from.” 

Non-academic  interactions  happened  also  when  a student,  intending  to  explain  his  delay  on 
coursework,  posted  message  telling  he  just  got  a new  baby.  It  was  observed  that  most  participants  responded 
with  a brief  “congratulation”  note.  When  asked  about  why,  PI  explained,  “I  don’t  know,  it’s  just  a formality  to 
congratulate  someone  getting  new  baby.”  During  the  course,  such  a formality-bounded  social  interaction  did 
occur  several  times. 

Finally,  in  the  chat-room  sessions  more  interpersonal  networking  messages  popped  up.  The 
comparison  of  bulletin  board  scripts  and  those  of  chat -room  indicated  a difference:  the  former  were  written  in  a 
more  formal  tone  while  the  latter  displayed  more  a personal  voice  and  mixed  with  more  social  presence  signs 
(such  as  “©”).  Some  participants  expressed  preference  of  chat-room  to  bulletin  board  because  “it  is  more  like 
face-to-face  talk”  while  the  others  believed  they  tended  to  think  more  during  bulletin  board  discussions. 

Managing  group  work:  Group  work,  in  participants’  perspectives,  involves  critical  management  issues 
of  timing  arrangement,  responsibility  specification,  communication  tools  selection,  and  investment  in  peer 
support. 

First,  online  observation  showed  that  timing  was  a critical  issue  in  group  work.  Some  participants 
displayed  a particular  concern  about  the  timing  within  teamwork.  On  the  group  discussion  forums,  these 
participants  tended  to  be  active  ones  who  initiated  the  negotiation  of  team  schedule.  Typically,  they 
volunteered  to  be  team  leaders,  with  an  intension  “to  get  it  done  and  out  of  it  anyway”.  They  posted  their 
speculations  of  the  timeline  to  the  whole  team.  They  softly  pushed  the  others  to  abide  by  the  schedule  by 
posting  the  messages  like,  “how  is  everything  going... we  have  only  one  week  left  for  the  finalization”  and  the 
sort.  Other  participants,  differently,  felt  more  comfortable  being  passive  and  pushed  by  the  peers.  One  said,  “I 
am  not  a very  organized  person  and  usually  wait  to  the  last  minute  to  do  my  work.  Group  is  a great  way  to 
monitor  and  push  me  onto  the  track.”  Due  to  such  a difference  of  the  timing  concept,  a tension  or  conflict  is 
inevitable  in  the  group.  Actually,  in  their  emails  to  the  instructor,  quite  a few  participants  complained  their 
teamwork  experiences  were  not  as  positive  as  they  expected,  because  “work  progress  was  delayed  as  teammates 
did  not  do  their  work  on  time” . 

Second,  some  participants  demanded  responsibility  specification  in  the  teamwork.  They  said: 

Having  labor  division  is  very  necessary.  Everybody  then  will  understand  what  to  do  and  when  to  do. 
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Usually,  when  I collaborate  with  people,  you  know,  especially  at  work,  we  do  good  job  defining,  you 
know,  evening  out  roles  and  responsibilities  at  the  beginning.  So  it  ends  out  being  fair,  you  know, 
break  out  the  work. 

They  hoped  that  by  specifying  responsibility,  there  would  be  “fairness’'  among  members’  investment  and 
commitment.  However,  they  also  expressed  a concern  on  the  binding  power  of  the  responsibility  specification, 
“Although  the  rules  were  laid  out,  people  would  not  interpret  them  as  the  same,  so  they  did  not  do  the  same.’’ 
Therefore,  some  participants  requested  that  the  instructor  should  have  played  a more  powerful  monitor  role, 

“He  should  be  able  to  tell  who  is  working  who  is  not.” 

Third,  the  participants  employed  specific  strategy  in  selecting  communication  tools  during  teamwork. 
P4  described: 

I use  group  discussion  forum  to  post  my  assignments,  ask  for  teammates’  opinions,  and  post  feedbacks 
to  teammates.  If  I need  to  talk  to  someone  privately,  I will  send  personal  email.  I would  like  to  use 
chat -room  for  instant  opinion  exchange,  but  it  is  so  difficult  to  schedule  a time  when  all  of  us  can  be 
online.  Then  I usually  email  the  instructor. 

This  description  was  echoed  by  the  researcher’s  observation  and  other  participants’  explanations. 

Finally,  the  participants  interpreted  their  investment  in  peer  support  as  an  intentional  action  of  “giving 
and  taking”  with  a sense  of  fairness.  For  instance,  P2  complained: 

I think  I’ve  given  more  than  I got  it  returned.  I mean  they  did  not  email  or  post  message  to  me  often 
enough.  Even  when  they  did  post  or  email  me,  they  did  not  write  enough.  I feel  a little  frustrated,  you 
know... 

Instructor-monitored  participation'.  Generally,  the  participants  favored  a participation  process  that  is 
monitored  and  supported  by  the  instructor,  whose  important  role  in  facilitating  the  online  discussion  was 
evident.  In  the  bulletin  board,  the  instructor’s  posting  attracted  the  most  responses.  A discussion  thread,  once 
replied  by  the  instructor,  usually  gained  a lot  more  responses  than  others.  This  might  be,  for  a major  part,  due 
to  the  fact  that  online  interaction  and  discussion  was  required  and  graded  in  the  course.  Responding  to  the 
instructor’s  message,  deemed  by  most  participants,  was  an  important  way  to  showing  presence  and 
involvement. 

Additionally,  the  instructor  was  regarded  an  expert  or  authority  in  subject  content.  P6  said,  “It  is  good 
to  have  multiple  opinions,  but  I feel  confused  as  to  which  one  to  pick.  I think  the  instructor  should  present  his 
view  so  we  will  know  which  one  is  correct.”  And  the  frequently  mentioned  expectations  of  the  instructor  were, 
“He  can  make  public  posting  to  everybody.” 

Finally,  in  interviewing,  the  participants  expressed  a strong  desire  that  the  instructor  should  specify 
assignment  timeline  and  monitor  students’  teamwork  progress.  For  example  : 

I know  that  we  have  assignment  timeline  in  this  course  content,  but  sometimes  it  is  a little  confusing  to 
put  them  together.  So  if  he  (the  instructor)  could  just  say  “here  is  where  we  are,  here  is  what  is  coming 
up  next,  just  to  give.,  .just  to  make  sure  everybody  is  in  the  same  track(P2). 

If  there  is  anything,  I think  he  should  monitor  people  with  the  things  we  are  doing,  ‘cause  right  now 
our  group  is  behind  the  schedule  for  the  one  who  was  sick  and  the  other  one  who  is  not  so  responsive. 

I guess  if  he  can  remind  people  to  work  on  time,  it  will  be  great  (P3). 

With  these  requests,  the  participants  preferred  the  frequent  monitoring  by  the  instructor.  This  finding  was  also 
evident  in  students’  emails  to  the  instructor:  the  most-often  inquired  topic  was  the  schedule  of  assignments. 

Self-adaptation:  Some  participants,  when  describing  their  online  learning  histories,  demonstrated  a 
process  of  personal  change  and  a want  for  self-adaptation.  P3,  a trainer  who  had  experienced  traditional  school 
education,  military  technical  training,  and  online  education,  said: 

Oh,  yeah,  it  is  definitely  a change.  Before  I went  to  the  Navy,  I was  accustomed  to  the  traditional 
classroom.  Then  when  I went  to  Navy  and  learned  different  type  of  training,  that  was  very 
concentrated,  very  fast.  And  then  now  I am  back  to  school  again  and  I am  doing  this  online.  And  this 
is  the  third  type  where  it  is  not  as  slow  in  the  classroom.  I feel  I get  more  comfortable  with  it  now. 
Similarly,  P8,  a self-claimed  solitary  learner,  said: 

I have  always  known  that  I love  individual  work  more  than  teamwork.  I realize  it  is  my  comfort  zone. 
But  I need  to  go  beyond  this  comfort  zone  and  take  challenge.  So  to  answer  your  question,  I will  say  I 
definitely  prefer  individual  work,  but  will  probably  choose  team  project. 

These  messages  reflected  that  these  adult  participants  were  willing  to  actively  adapt  themselves  to  different 
learning  contexts  and  demands. 
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Affective  Response 

The  affective  response  involves  participants’  perceptions  and  feelings  developed  through  their  learning 
processes,  including:  unwillingness  to  be  interdependent,  sense  of  unfairness,  sense  of  responsibility,  and 
satisfaction. 

Unwilling  to  be  interdependent:  Some  participants  obviously  displayed  a reluctance  to  rely  on  other 
members  of  the  team  in  completing  learning  tasks.  P2  said, 

1 tend  to  know  what  my  ability  and  capability  levels  are,  and  1 would  rather  rely  on  just  myself  to 
accomplish  an  assignment  verses  the  other  people,  because  I’ve  been  disappointed  so  far  (sigh)...  My 
time  is  limited.  It  is  very  difficult  to  depend  on  some  other’s  schedule.  I have  to  work  ahead.  If  I have 
to  wait  for  others’  pieces  before  I can  do,  I cannot  work  ahead.  It  is  difficult  for  me. 

Her  unwillingness  to  be  interdependent  might  be  due  to  several  motives:  trust  on  personal  ability  rather  than 
others;  former  teamwork  experiences  being  disappointing;  and  timing  concern.  Like  P3,  P4,  and  P5,  she  felt 
relying  on  herself  is  the  most  “secured”  way  to  feel  “more  at  control”. 

Sense  of  unfairness:  The  participants  were  sensitive  to  peers’  different  levels  of  participation  and 
interaction.  They  read  this  difference  as  a sense  of  unfairness.  For  instance,  P5  and  P8  complained,  “People 
participate  at  different  levels,  you  know.  I am  in  the  middle,  in  terms  of  posting  and  giving  feedbacks.”  P2 
noticed  that  “some  persons’  messages  getting  more  feedbacks.  I usually  respond  to  my  teammates’  messages, 
but  when  I posted,  there  was  not  much  returned.”  Pressed  by  their  sense  of  unfairness,  the  participants 
demanded  a clear  labor  division  in  the  teamwork: 

PI:  Having  labor  division  is  very  necessary.  Everybody  then  will  understand  what  to  do  and  when  to 
do. 

P2:  So  it  ends  out  being  fair,  you  know,  break  out  the  work. 

As  a result  of  sense  of  unfairness,  most  participants  expressed  “feeling  disappointed”  and  did  self-blaming:  “I 
don’t  know.  Maybe  what  I wrote  was  not  interesting  enough  (P2),”  “I  guess  it  is  my  language.  People  cannot 
understand  my  English.  I need  to  make  my  ideas  clearer  (P10),”  and  “I  guess  it  is  because  of  my  background.  I 
found  most  students  in  this  class  are  teachers,  I am  not  (P3).”  Generally,  they  found  themselves  “spending  less 
time  on  online  activities  now  and  focus  on  my  offline  reading  and  completing  assignment.” 

Sense  of  responsibility:  When  asked  about  their  roles  in  online  participation  and  collaboration, 
participants  kept  mentioning  “responsible”  or  “responsibility”: 

P2: 1 followed  all  the  instructions  that  were  laid  out  regarding  the  responsibilities  as  the  project  leader. 
P3: 1 think  I have  a strong  sense  of  responsibility.  Individually,  I may  wait  to  the  deadline  to  do  my 
work.  But  in  teamwork,  I have  to  do  it  right  ‘cause  others  need  my  work  to  continue.  And  I just  don’t 
want  to  let  them  down. 

P4:  The  difference  between  individual  and  group  work  is  by  individual,  I am  responsible  for  my 
schedule  and  my  progress;  by  group,  I am  also  responsible  for  my  teammates’  progress. 

P5:  Being  responsible  is  to  do  the  work  on  time  and  giving  feedback  to  your  teams.  At  least,  do  what 
the  syllabus  required. 

P7  and  P10:  Responsible  teammates  are  necessary  for  good  collaboration. 

P12: 1 did  ask  them  why  (they  were  late  in  doing  work),  like,  “How  come  you  did  not  post”.  I think 
that  is  not  my  responsibility. 

Sense  of  responsibility,  as  the  researcher  observed  and  interpreted,  was  the  most  important  factor  fostering  the 
appropriate  behaviors  in  online  learning  participation  and  collaboration.  Being  irresponsible  was  understood  as 
“personality  issue”  (P3).  An  interesting  point  to  note  was  one  participant,  who  had  been  regarded  by  peers  as 
not  responding  and  being  delayed  in  coursework  completion,  also  believed  responsibility  was  necessary,  “Yeh, 
of  course  I think  responsibility  is  important.  I will  do  what  I am  expected  to  do.”  Reluctance  to  admit  being 
irresponsible  seems  natural. 

Satisfaction:  During  interviewing,  most  participants  explained  they  were  satisfied  with  their  present 
learning  processes  in  INSYS  446.  Positive  view  on  the  instructor  seemed  to  be  the  top  ground  for  the 
satisfaction: 

P2: 1 think  he  is  a good  instructor.  I think  that. . .he  is  very  available.  I did  not  like  my  first  one  (online 
learning  course),  I did  not  think  the  instructor  was  that  great. 
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P3: 1 think  he  is  good.  He  is  getting  back  right  away  in  responding  email.  If  there  is  anything,  I think 
he  should  monitor  people  with  the  things  we  are  doing. 

P5:  Sometimes  we  have  very  open-ended  questions,  like  what  we  had  in  reading  review.  Sometimes 
we  have  a checklist  to  review  what  we  should  pay  attention  to  in  project.  I think  this  is  a good  balance. 
P6:  He  usually  gives  detailed  feedback  to  our  assignments,  which  is  really  great. 

Then,  participants  expressed  welcome  to  their  learning  tasks: 

P7:  There  is  a clear  guideline  on  what  we  should  do  for  the  project,  but  it  also  gives  us  space  for  free 
thinking. 

P8: 1 like  the  first  two  weeks’  general  discussion  (on  readings)  very  much,  I feel  I have  learned  most 
from  that. 

PI  1:  I like  my  project.  It  is  my  problem  in  the  work.  I will  teach  my  people  how  to  use  Excel  as  a 
management  tool. 

P12:  I like  the  way  we  did  our  learning.  First  we  read  and  gained  necessary  knowledge,  then  we  had 
hand-on  experiences.  I think  this  is  reasonable. 

Finally,  the  participants  valued  collaboration  opportunities  in  the  online  course.  They  felt  they  could 
test  and  enrich  their  thoughts  in  teamwork  and  group  discussion: 

P2:  The  nature  of  the  project  was  that  it  did  require  participation  from  the  other  members’  in  the  group 
just  so  that  we  can  get  feedback  on  how  our  mindtool  was  being  used,  so  if  I were  to  do  it  myself,  it 
would  be  really  hard  to  get  feedback  from  somebody  who  understand  what  I was  doing. 

P5: 1 think  when  I am  working  with  the  group,  and  I am  having  that  feedback  back  and  forth,  it  lists 
more  information  pulled  more  out  of  me  and  I get  more  information  from  other  sources  besides  what  I 
have,  so  I get  more  knowledge  than  what  is  in  my  head,  more  experiences  that  other  people  might  have 
that  I wouldn’t  have  had  myself. 

The  above  statements  indicated  that  the  participants  involved  in  collaboration  for  sharing  perspectives  with 
fellow  students.  Getting  informative  feedback  and  expertise  exchange  attracted  them  into  group  work. 

Discussion 

Conclusively,  in  the  case  studied,  the  participants  share  similar  responses  to  communication,  action, 
and  participation  in  terms  of  purposes  and  rules;  they  implement  communication  tools  to  collaboratively  learn; 
and  they  prefer  to  experience  a division  of  community  labor  or  responsibility.  These  three  operations  agree 
with  the  activity  theory  which  Hung  and  Chen  (2002)  have  proposed  as  a framework  for  learning  community 
operation:  rules,  tools,  and  division  of  labor  as  three  bonds  to  affiliate  subjects  with  the  learning  community. 

Then,  in  this  case  the  participants  interviewed  and  observed  have  not  presented  enough  evidences 
showing  a strong  sense  of  connection  or  belonging  (McMillan  & Chavis,  1986).  Actually,  the  participants  did 
not  value  the  investment  of  timing  and  efforts  into  relationship  construction,  mainly  due  to  timing  concern  and 
doubt  on  the  networking  purpose.  This  finding  echoes  Brown’s  (2001)  study  conclusion:  The  participants  did 
not  perceive  community  to  exist  online  and  they  did  not  place  a high  priority  on  devoting  time  to  fostering 
relationships.  However,  a sense  of  relevancy  do  exist  and  help  to  create  a degree  of  bonding:  participants  tend 
to  respond  to  familiar  names  in  bulletin  board;  participants  prefer  views  from  people  who  share  similar  career 
background;  and  participants  who  have  different  background  from  the  majority  deem  themselves  as  more 
“outsider”  (P2  and  P9). 

Additionally,  the  feeling  of  trust  (Rovai,  2002)  is  not  realized  in  this  case.  Most  participants  are 
unwilling  to  rely  on  other  members  of  the  team.  Reasons  may  be  the  frustrating  teamwork  experience  from 
previous  online  learning,  the  individual  difference  in  perceptions  of  responsibility  and  learning  habits 
(organized  or  not),  and  the  lack  of  familiarity  of  peers.  One  week’s  online  introduction  cannot  construct  trust 
across  the  distance. 

Based  on  these  findings,  the  researcher  tends  to  believe  that  in  spite  of  the  instructor’s  community 
initiative  in  instructional  context  design,  a sense  of  community  has  not  been  developed  among  students.  As 
Lock  (2002)  argues,  a community-promoting  virtual  learning  environment  will  not  ensure  the  forming  of 
community.  It  is  students’  initiative  and  active  participation  cultivates  the  forming  of  virtual  learning 
community. 

Some  particular  pedagogical  issues,  as  this  case  reveals,  should  be  taken  care  of  in  order  to  assist 
students  to  make  a shift  into  community  philosophy  and  actions: 

o Awareness  of  a learning  community  framework  at  the  inception : The  participants  should  be  aware  of 
community  philosophy  and  how  it  works  at  the  inception  period.  As  Shapiro  and  Levine  (1999) 
recommend,  students  need  to  be  open  and  to  be  willing  to  reframe  their  roles  as  learners.  Lock  (2002) 
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also  proposes,  “It  is  the  informed  initiative  of  members  and  the  leadership  of  the  community  that 
influence  and  foster  and  sustain  the  vibrancy  and  resiliency  of  an  online  learning  community”  (p.  406). 
Designers  and  developers  of  a virtual  learning  community  need  to  build  students’  awareness  of 
community  model  and  have  support  structures  in  place  to  foster  their  shift  in  thinking  and  behaviors. 
For  instance,  showing  role  models  and  explicitly  explaining  appropriate  learning  strategies  will 
encourage  students  to  display  more  investment  and  commitment  into  peer  support  within  community 
context. 

o Constructing  guidelines  and  norms  on  communication  and  participation : Clarification  of  expected 
individual  behavior  and  responsibility  is  necessary  when  planning  a virtual  learning  community.  As 
Lock  (2002)  explained,  the  construction  of  norms  can  be  first  directed  by  instructor  then  modified  or 
developed  by  students. 

o The  instructor's  role  and  heavy  load:  As  demonstrated  by  the  findings  in  this  case,  instructor  needs  to 
play  an  important  role  (as  a monitor,  facilitator,  and  expert)  in  the  learning  process,  even  though  the 
community  approach  is  more  student-centered.  The  instructor’s  presence  and  instant  feedback  are  the 
most  active  catalyst  to  support  online  participations.  However,  this  usually  means  a heavy  load  on  the 
instructor  part.  A strategy  to  reduce  instructor’s  load  may  be  the  leadership  development:  students 
who  play  the  leadership  role  are  able  to  initiate  and  organize  the  community  learning  voluntarily. 

o Small  group  verse  class  community:  It  is  also  noticed  that  in  this  case  participants  have  a stronger 
sense  of  interaction  with  peers  in  their  groups  rather  than  the  general  class  community.  This  kind  of 
task-oriented  group-based  interactions,  to  some  degree,  has  hindered  the  development  of  sense  of 
connection  to  the  class  community. 

Suggestions  for  Future  Research 

Bounded  by  timing,  this  case  study  has  not  done  a follo\wup  data  collection  of  participants’ 
perspectives  and  activities  beyond  the  course.  As  Brown  (2001)  indicates,  participants  may  be  involving  in 
long-term  interactions  that  are  beyond  the  course  cycle.  Therefore,  a follow-up  study  on  people  who  involve  in 
long-term  learning  affiliations  with  others  may  be  desirable.  In  addition,  an  evaluation  research  on  measuring 
the  development  stages  of  virtual  learning  community  construction  is  also  necessary.  Finally,  how  can  the 
leadership  role  be  supported  and  nurtured  with  the  purpose  of  fostering  virtual  learning  community 
development  is  also  an  important  research  question  to  be  explored. 
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Introduction 

Pre -service  teacher  programs  have  a central  goal  of  fostering  skills  necessary  for  students  to  become 
successful  professional  educators.  This  goal  is  often  challenging  because  students  typically  do  not  have 
sufficient  opportunities  to  gain  realistic  experiences  before  becoming  teachers  (Andrews,  2002).  In  addition, 
teaching  is  a complex,  dynamic  profession  where  challenges  regularly  occur  which  require  teachers  to 
incorporate  new  information,  make  decisions,  and  problem-solve  on  a regular  basis  (Howard,  2002;  Jonassen  & 
Hernandez-Serrano,  2002). 

Teaching  students  with  disabilities  represents  one  such  challenge  to  general  education  teachers.  For 
example,  the  1997  Individuals  with  Disabilities  Education  Act  (IDEA)  requires  that  students  with  disabilities 
receive  a free,  appropriate  public  education  (FAPE)  in  the  least  restrictive  environment  (LRE).  The  LRE  is  an 
environment  that  best  meets  students  ’ needs  in  the  most  typical  educational  setting.  This  requirement  has 
resulted  in  a dramatic  change  in  the  way  students  with  disabilities  are  educated.  A majority  of  students  are  now 
educated  in  typical  general  education  classrooms  with  special  education  supports  (Hallahan  & Kauffman,  2003; 
Turnbull,  Turnbull,  Wehmeyer,  & Park,  2003).  This  change  has  meant  that  general  education  teachers  need  to 
have  sufficient  background  knowledge  and  experience  with  identifying  students  that  may  need  to  be  assessed 
and  referred  to  special  education  services.  This  challenge  may  be  compounded  for  general  education  teachers 
who  are  working  with  students  who  are  English  Language  Learners  (ELL).  These  educators  may  lack 
experience  distinguishing  between  academic  difficulties  resulting  from  second  language  acquisition  and  those 
resulting  from  learning  disabilities  (Ochoa,  Gerber,  Leafstedt,  Hough,  Kyle,  Rogers -Adkinson,  & Koomar, 
2001). 

A further  requirement  of  IDEA  mandates  that  teachers,  parents,  students  (when  appropriate),  and 
support  personnel  (i.e.  school  psychologist,  principal)  meet  to  develop  an  Individualized  Education  Program 
(IEP).  The  IEP  is  a planning  process  that  results  in  a signed  legal  document  that  specifies  a student’s  academic 
skills,  needs,  and  goals  and  acts  as  a guide  for  academic  instruction  for  the  student  (Hallahan  & Kauffman, 
2003).  This  process  requires  an  understanding  of  special  education  law,  as  well  as  skills  to  work  collaboratively 
with  a diverse  group  of  individuals  with  different  goals  to  reach  a decision. 

Pre -service  teachers  require  extensive  knowledge  and  experience.  But,  it  is  neither  practical,  nor  in  the 
best  interests  of  students  with  disabilities,  to  train  pre  -service  teachers  solely  in  a field-based  classroom 
(Andrews,  2002).  Opportunities  for  contextualized  learning  beyond  observation  and  practicum  experiences  are 
important  to  support  pre -service  learning  (Baker,  2000).  One  such  approach  is  problem-based  learning  (PBL). 
PBL  is  an  instructional  technique  in  which  meaningful  tasks,  often  in  the  form  of  problems,  serve  as  the  context 
and  stimulus  for  knowledge  building  and  critical  thinking  (Howard,  2002).  Situations  used  in  PBL  are  often 
what  are  called  “ill-structured”  (Jonassen  & Hernandez-Serrano,  2002)  and  typically  mirror  real  life  decisions 
that  professionals  (i.e.  educators,  doctors)  need  to  make  based  on  incomplete  and  constantly  changing 
information  that  often  don’t  have  a clearly  defined  solution  (Baker,  2000;  Howard,  2002). 

This  study  researched  the  implementation  of  a multimedia  PBL  module,  entitled  Multicultural  Special 
Education  (MUSE),  in  a contextualized  learning  experience.  The  MUSE  module  was  developed  as  part  of  the 
CASELINK  series  of  multimedia  PBL  cases  designed  by  researchers  at  the  University  of  California  at  Santa 
Barbara  to  train  pre  -service  teachers  to  think  about  special  education  issues  in  a realistic,  professional  context 
(Gerber,  English,  & Singer,  1999).  The  main  goal  of  the  modules  is  to  give  pre -service  teachers  an  opportunity 
to  use  information  and  interact  with  their  peers  in  such  a way  that  they  become  self-sufficient,  life  long  learners 
that  are  able  to  adapt  to  new  professional  situations  (Ochoa,  Kelly,  Stuart,  & Rogers -Adkinson,  2004). 

The  MUSE  module  in  this  study  was  utilized  as  part  of  a course  designed  to  introduce  elementary 
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education  majors  to  special  education  and  teaching  exceptional  learners.  The  module  required  pre  -service 
teachers  to  assess  a student  named  “Andres”  who  has  limited  English  proficiency  (LEP)  and  potential  learning 
disabilities,  to  participate  on  a team  to  create  an  IEP,  and  make  decisions  about  whether  or  not  to  refer  Andres 
to  special  education  (Ochoa  et  al,  2004).  The  module  provided  information  about  the  student  through  interviews 
with  school  personnel  and  family  representatives,  video  footage  of  actual  school  interactions,  artifacts  of  the 
student’s  work  (i.e.  writing  samples,  drawings,  test  scores)  and  background  about  appropriate  special  education 
laws.  In  addition,  pre  -service  teachers  were  required  to  role  -play  a typical  IEP  member  during  the  decision- 
making process  to  encourage  realistic,  collaborative  problem-solving.  Roles  included  a school  principal,  a 
special  education  teacher,  a parent,  a classroom  teacher,  a bilingual  education  teacher,  and  a school 
psychologist.  Students  were  provided  with  video  and  audio  interviews,  documents,  and  Internet  web-links  that 
described  each  role’s  perspective. 

Research  questions 

1)  Are  pre -service  elementary  school  teachers  enrolled  in  an  introductory  special  education  course 
satisfied  that  engaging  in  the  PBL  activity  prepares  them  to  be  professional  educators? 

2)  What  are  ways  in  which  PBL  fosters  engagement  with  special  education  concepts  and  practices  for 
pre -service  teachers? 

3)  How  does  the  use  of  multimedia  in  the  module  impact  satisfaction  with  the  PBL  experience? 

Method 


Participants 

Thirty-three  students  were  enrolled  in  an  introductory  course  on  teaching  exceptional  learners  in 
elementary  education  at  a large  Midwestern  university.  The  students  were  required  to  complete  the  MUSE 
module  as  part  of  the  requirements  for  the  course.  Twenty-nine  students  were  female  and  four  students  were 
male.  Thirty-one  students  were  Caucasian,  one  was  Asian,  and  one  was  Latino.  Ten  students  were  seniors, 
nineteen  were  juniors,  and  four  were  sophomores.  Twenty -nine  students  were  elementary  education  majors  and 
four  were  non-education  majors.  The  majority  of  students  had  some  practicum  or  field  experience  prior  to 
enrolling  in  the  class,  but  the  majority  of  students  indicated  that  they  had  little  or  no  experience  working  with 
students  with  disabilities.  Students  also  had  no  experience  evaluating  a student,  creating  an  IEP,  or  using  a PBL- 
based  case.  In  order  to  complete  the  module,  students  worked  for  six  75-minute  class  periods  in  groups  of  five 
to  six  to  assess  the  case  student,  explore  the  problem  from  different  roles/perspectives,  create  an  educational 
plan,  and  make  a decision  about  whether  or  not  the  student  should  be  referred  to  special  education. 

Setting 

The  study  was  conducted  at  a large  Midwestern  research  level  one  university.  The  instructor  was  a 
second  year  doctoral  student  with  diverse  experiences  working  with  students  with  disabilities.  She  had  taught 
this  course  twice  previously  and  used  the  MUSE  module  both  times.  She  was  also  one  of  the  researchers  in  this 
study. 

During  the  PBL  activity,  the  class  was  grouped  into  six  self-formed  groups.  The  activities  took  place 
over  the  course  of  six  sessions.  The  first  and  last  sessions  took  place  in  the  general  classroom  and  the  remaining 
four  took  place  in  a computer  lab  equipped  with  thirty-five  computers.  The  first  session  provided  an 
introduction  to  the  PBL  process.  The  instructor  gave  a brief  overview  of  PBL,  had  students  viewed  excerpts  of 
problem-solving  strategies  fromthe  movie  Apollo  13  and  discussed  problem  based  learning  concepts. 

Over  the  next  four  classes  in  the  lab,  the  pre -service  teachers  participated  in  three  activities  that 
involved  both  individual  and  group  work.  The  first  activity  was  an  introduction  to  the  student  and  required 
participants  to  write  a brief  assessment  based  on  the  information  provided  in  the  case.  The  second  activity 
involved  individuals  investigating  the  scenario  more  in-depth  and  developing  a group  consensus  about  the 
student’s  situation.  The  third  activity  involved  participants  selecting  a role  strand  and  exploring  information 
from  only  one  role’s  perspective  and  deciding  on  an  IEP  goal  for  the  student.  The  small  groups  then  met, 
compared  information  and  developed  a common  educational  plan  through  a process  of  negotiating,  discussion, 
and  problem-solving.  On  the  final  day,  students  met  again  in  the  regular  classroom  in  their  small  groups  to 
develop  final  recommendations  on  whether  the  student  should  be  referred  to  special  education  or  not,  and  then 
participated  in  a class-wide  discussion  about  their  recommendations  and  their  overall  thoughts  about  the  PBL 
module. 


444 


Procedures 

The  study  used  both  qualitative  and  quantitative  methodologies  to  collect  and  analyze  the  data.  This 
mixed  methodology  approach  was  not  only  useful  in  understanding  the  student’s  learning  but  also  allowed  the 
opportunity  to  triangulate  and  verify  both  quantitative  and  qualitative  results. 

Quantitative  data  was  collected  from  a satisfaction  and  perception  survey  (see  Table  1)  developed  by 
the  researchers  to  gain  participant  feedback  on  our  research  interests:  satisfaction  with  the  level  of  professional 
preparation  from  the  PBL  module,  perceived  knowledge  acquisition  of  special  education  concepts,  and 
satisfaction  with  the  multimedia  component  of  the  PBL  module.  Survey  questions  were  formulated  after  two 
observations  by  researchers  that  gathered  preliminary  data  about  module  usage  and  group  interactions. 

A 15-item  questionnaire  was  administered  to  pre-service  teachers  after  the  completion  of  all  MUSE 
module  activities  (see  Table  1).  Twenty-nine  students  (n=29)  were  in  attendance  the  day  the  survey  was 
administered  and  all  students  responded.  Two  separate  metrics  were  used  in  the  survey.  Questions  1-13 
utilized  a five-point  Likert  scale  which  measured  relative  strength  of  agreement,  while  questions  14-15  utilized 
a 4-point  Likert  scale  measuring  relative  frequency.  Scores  were  standardized  by  converting  them  to  z scores. 
Some  questions  were  worded  in  a manner  such  that  the  metric  measured  low  satisfaction,  therefore  it  was 
necessary  to  reverse-scale  items  representing  an  opposite  dimension.  Reverse-scaled  items  included  4,  8,  10,  12, 
13.  All  subsequent  calculations  used  standardized  scores  and  appropriate  reverse-scaled  items. 

The  statistical  package  SPSS  was  utilized  to  calculate  an  internal  consistency  estimate  of  reliability  on 
the  survey  questions  (Green  & Salkind,  2002).  Each  item  of  the  survey  was  assumed  to  be  approximately 
equivalent  to  every  other  item  and  questions  were  all  designed  to  measure  participant  response  to  the  MUSE 
module.  It  was  also  assumed  that  any  errors  in  measurement  between  questions  were  unrelated.  There  was  no 
time  limit  and  questions  were  syntactically  different  in  order  to  reduce  the  likelihood  that  measurement  errors 
were  related,  as  in  the  case  where,  for  example,  someone  had  circled  all  ‘l’s.  Lastly,  it  was  assumed  that  the 
internal  consistency  estimate  would  accurately  reflect  the  scale’s  reliability. 

Two  measures  were  computed:  a coefficient  alpha  and  a split -half  coefficient  expressed  as  a 
Spearman-Brown  corrected  correlation.  For  the  split-half  coefficient,  the  scale  was  split  into  two  halves  in  a 
manner  such  that  the  two  halves  would  be  as  equivalent  as  possible.  In  splitting  the  items,  the  sequencing  of  the 
items  as  well  as  whether  or  not  items  assessed  participant  satisfaction  were  taken  into  account.  The  first  half 
included  items  1,3, 5, 7, 9,1 1,13,15,  while  the  other  half  included  items  2,4,6,8,10,12,14.  Values  for  coefficient 
alpha  and  the  split-half  coefficient  were  .60  and  .74,  respectively,  indicating  moderate  reliability  demonstrated 
by  the  coefficient  alpha  and  satisfactory  reliability  demonstrated  by  the  split-half  coefficient. 


Table  1 Andres  Case  Survey 


Descriptive  Statistics 

K 

Minimum 

Maximum 

Mean 

Std.  Deviation 

1. 1 learned  a lot  about  the  student  in  the  case. 

m 

1.00 

4.00 

2.83 

.89 

2. 1 felt  invested  in  my  team. 

m 

1.00 

3.00 

2.14 

.44 

3. 1 feel  better  equipped  to  make  a similar  decision 
in  real  life  after  doing  this  PBL  module. 

29 

1.00 

5.00 

2.52 

.83 

4. 1 did  not  have  enough  information  from  the 
module  to  effectively  make  a decision  on  the  case. 

29 

1.00 

4.00 

1.86 

.69 

5.  The  media  (pictures,  movies,  sounds,  text) 
helped  me  to  understand  the  content  of  the  module. 

29 

1.00 

3.00 

1.97 

.57 

6. 1 feel  I could  be  on  a real  student's  team  now  as 
a result  of  this  case. 

29 

2.00 

5.00 

3.04 

.68 

7.  The  media  (pictures,  movies,  sounds,  text) 
helped  me  feel  more  connected  to  the  case  than  if  it 
had  been  text  alone. 

29 

1.00 

4.00 

1.55 

.69 

8. 1 wasn't  able  to  learn  a lot  from  the  case 
materials. 

29 

2.00 

4.00 

3.52 

.79 

9.  This  module  gave  me  a sense  of  a real  student's 
situation. 

29 

1.00 

5.00 

2.41 

.87 

10. 1 ignored  the  media  (pictures,  movies,  sounds, 
text). 

29 

4.00 

5.00 

4.55 

.51 

11.1  felt  an  emotional  attachment  to  the  student  in 
the  module. 

29 

2.00 

5.00 

3.28 

.88 
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12. 1 ignored  the  hyperlinks  to  external  websites. 

■ 

2.00 

4.00 

2.72 

13.1  am  not  likely  to  have  a situation  like  this  in 
my  own  classroom. 

29 

3.00 

5.00 

4.31 

.60 

14.  How  often  did  you  refer  to  information  you  had 
learned  earlier  in  the  case  during  the  final  group 
decision? 

29 

1.00 

2.00 

1.48 

.51 

15.  How  often  did  you  use  the  hyperlinks  to 
external  websites  to  gain  more  information? 

29 

1.00 

4.00 

2.79 

.98 

Valid  N (listwise) 

m, 

The  study  utilized  a variety  of  qualitative  data  sources  to  evaluate  the  research  questions  in  order  to 
triangulate  the  findings  and  increase  the  internal  validity  of  the  data.  Data  sources  included  student  reflections 
submitted  to  the  instructor  as  part  of  the  general  module  requirements,  an  interview  with  a student  participant, 
and  observations  of  the  collaboration  process.  The  constant  comparative  method  (Glaser  & Strauss,  1967)  was 
used  to  evaluate  transcripts  and  documents  and  identify  emergent  themes  from  the  data. 

The  qualitative  data  came  from  three  sources:  student  reflections  during  the  case,  an  open-ended 
question  on  the  survey,  and  an  interview  transcript.  Students  posted  reflections  to  online  questions  viewed  only 
by  the  instructor  after  the  completion  of  the  second  and  third  activities  during  the  module.  They  were  asked  to 
respond  to  the  following  prompts: 

Reflect  on  the  Andres  activities  up  to  now.  What  are  some  of  your  thoughts,  concerns,  questions,  or 
issues  that  come  to  mind?  Think  about  what  you  're  exploring  related  to  Andres  as  well  as  the  Problem 
Based  Learning  (PBL)  process  itself. 

After  the  final  session,  students  were  required  to  post  a response  to  the  following  prompts: 

One  of  the  major  objectives  of  this  activity  was  to  simulate  the  dynamics  of  the  interdisciplinary  team. 
What  were  some  of  the  challenges  your  group  experienced?  How  did  you  resolve  them?  Respond  to 
these  two  questions.  Also,  provide  any  additional  overall  comments  on  this  activity. 

Participants  were  also  given  a space  to  respond  to  an  open-ended  question  on  the  survey.  The  questions 
asked  students  to  “Please  add  any  additional  comments  about  the  Andres  activity.” 

Results  and  Discussion 

In  analyzing  the  data,  we  found  several  recurrent  themes  that  characterized  the  pre -service  teachers’ 
experiences  using  the  multimedia  PBL  module.  We  have  categorized  the  themes  within  the  framework  of  each 
research  question. 

Question  1:  Are  pre-service  elementary  school  teachers  enrolled  in  an  introductory  special  education  course 
satisfied  that  engaging  in  the  Problem  Based  Learning  activity  prepares  them  to  be  professional  educators? 

LA)  The  experience  had  real-life  applicability 

The  sense  of  authenticity  about  the  characters  and  team  planning  extended  to  the  ability  of  the  learning 
experience  to  have  real-life  applicability  for  the  participants.  The  PBL  process  involves  participants  in  solving 
‘real-life’  problems  from  which  knowledge  and  experience  can  be  gained  (Bridges  & Hallinger,  1997).  The 
MUSE  participants  were  enthusiastic  in  their  opinions  of  the  experience  and  the  sense  of  preparation  they  felt  as 
a result  of  their  participation  in  the  process.  Three  primary  areas  of  real-life  applicability  included  a sense  of 
better  understanding  the  requirements  of  teaching  students  with  special  needs  and  second  language  learners, 
being  more  prepared  to  assess  a student,  and  participating  on  an  IEP  team. 

Participants  expressed  the  opinion  that  they  gained  a realistic  sense  of  what  it  would  be  like  to  have  a 
student  with  a disability  or  an  English  as  a Second  Language  learner  in  their  classrooms.  The  MUSE  case  gave 
them  the  opportunity  to  explore  materials  about  the  student  through  observations  of  his  interactions  in  the 
school,  samples  of  his  academic  work,  and  comments  about  the  student  by  school  personnel  and  family 
representative.  After  interacting  with  the  material,  they  were  able  to  discuss  the  content  and  their  perspectives 
with  teammates,  and  make  decisions  based  on  what  they  viewed.  Participants  commented  that  this  process  gave 
them  an  opportunity  to  get  to  know  how  to  better  meet  the  needs  of  students  in  their  future  classrooms.  For 
example,  participant  comments  included: 
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The  Andres  case  has  given  me  a good  perspective  what  it  would  be  like  to  have  a child  in  the 
classroom  that  needs  special  help  and  also  struggles  with  the  English  language. 

I believe  I learned  a lot  and  I now  realize  how  hard  it  is  to  be  a teacher  deeding  with  a student 
like  Andres  and  the  time  and  effort  that  goes  into  working  with  this  child. 

For  most,  if  not  all,  of  the  participants,  this  was  the  first  time  they  had  exposure  to  the  IEP 
process.  This  finding  also  was  the  case  for  the  student  interviewed: 

I think  it  helped  to  an  extent.  I did  not  really  know  what  an  IEP  was  before.  So  at  least  I am 
aware  of  what  it  is,  what  the  expectations  are,  and  what  to  go  about  making  one. 

In  addition  to  feeling  a sense  that  they  gained  insight  into  having  a student  with  special  needs  in  the 
classroom,  the  students  also  expressed  that  they  felt  better  prepared  to  assess  whether  or  not  a student  had  a 
disability.  This  particular  notion  is  not  often  held  by  novice  teachers,  especially  within  general  education 
(Stough  & Palmer,  2003).  This  challenge  is  particularly  compounded  when  a student’s  first  language  is  different 
than  the  teacher’s  because  it  may  be  difficult  to  assess  whether  learning  difficulties  are  a result  of  not 
understanding  the  language  of  instruction  or  from  a disability  (Rogers -Adkinson,  Ochoa,  & Delgado,  2003). 

The  participants  were  part  of  a team  that  explored  what  they  each  knew  about  the  student  and  then  made 
decisions  about  the  student’s  academic  needs.  The  pre-service  teachers  expressed  that  they  gained  an  increased 
understanding  of  the  assessment  process  through  their  activities  in  the  MUSE  case.  Typical  comments  included: 

I have  really  enjoyed  this  exploration.  I have  learned  many  things  that  will  help  me  be  able  to 
evaluate  students  who  might  need  special  help.  I have  really  enjoyed  the  group  activities. 

Referral  is  a really  tough  job  to  do  and  as  a future  educator  you  want  to  do  what  is  best  for 
the  child.  It  was  really  hard,  but  on  the  other  hand,  it  gave  some  really  great  experience  on 
the  evaluation  process. 

l.B)  Participants  developed  collaboration  skills 

In  school  settings,  collaboration  skills  are  essential  (Tschannen -Moran  & Woolfolk-Hay,  2000). 
General  education  teachers  must  be  skillful  in  working  with  special  education  teachers,  parents/caregivers,  and 
other  school  professionals  in  order  to  successfully  meet  the  needs  of  their  students  (Gerber  et  al.,  1999; 
Matthews  & Menna,  2003).  Collaboration  is  often  a skill  that  is  not  taught  enough  in  teacher  education 
programs  but  is  required  more  and  more  due  to  educational  reform  efforts  (Tschannen -Moran  & Woolfolk-Hay, 
2000).  In  addition,  those  involved  in  the  decision  process  represent  different  goals  and  knowledge  bases  - all  of 
which  must  come  together  to  insure  that  students  are  being  taught  the  goals  identified  in  the  IEP  in  the  Least 
Restrictive  Environment  (Gerber  et  al.,  1999;  Howard,  2002). 

The  MUSE  module  provided  numerous  opportunities  for  participants  to  have  discussions,  negotiate 
and  work  together  with  a team.  Team  members  each  took  on  a typical  role  of  an  IEP  meeting  participant  (i.e. 
teacher,  parent,  psychologist)  and  contributed  their  information  to  the  team  from  that  perspective.  Once 
participants  shared  their  information,  teams  were  required  to  make  decisions  about  the  student.  This  process  of 
information  sharing  and  decision-making  provided  students  an  opportunity  to  learn  to  collaborate  to  accomplish 
their  goals.  Student  reflections  included  numerous  comments  about  the  collaboration  process: 

This  activity  has  been  very  successful  in  working  as  a group  with  other  people  in  our  class. 

We  have  learned  so  much  as  a group  in  how  to  solve  problems  in  the  education  field  with  this 
real  situation  regarding  Andres. 

It  was  in  this  exercise  that  we  really  came  together  as  a group,  all  offering  our  own  insight 
into  what  would  be  the  best  assessments  for  Andres,  as  well  as  important  goals  for  him. 

Question  2)  What  are  ways  in  which  PBL  fosters  engagement  with  special  education  concepts  and  practices  for 
pre-service  teachers? 
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2.  A)  PBL  provides  an  authentic  context  for  learning 

One  of  the  goals  of  PBL  is  to  provide  an  authentic  learning  experience  for  participants  (Albanese  & 
Mitchell,  1993).  This  goal  is  particularly  important  for  pre  -service  teachers  who  are  working  to  gain  skills  in 
order  to  make  them  more  effective  teachers  (Bridges  & Hallinger,  1997).  Issues  of  meeting  the  needs  of 
students  with  special  learning  needs  are  of  particular  concern  for  pre  -service  teachers  (Ochoa,  Vasquez,  & 
Gerber,  1999).  Since  each  student’s  individual  needs  differ  greatly,  they  require  individual  attention  and  unique 
interventions  to  tailor  a plan  to  create  the  best  environment  for  their  learning  (Andrews,  2002).  Their  needs 
require  the  ability  to  use  general  educational  concepts  in  a variety  of  situations.  One  particular  challenge  for 
teachers  of  students  with  limited  English  proficiency  (LEP)  is  distinguishing  between  issues  of  second  language 
acquisition  and  learning  disabilities  (Ochoa  et  ah,  1999).  The  MUSE  module  offered  pre  -service  teachers  the 
opportunity  to  engage  in  a learning  experience  that  explores  the  process  of  assessing  the  needs  of  a student  with 
LEP  and  a potential  learning  disability  and  then  participating  on  a team  to  address  his  learning  needs. 

From  the  comments  expressed  by  the  participants,  there  is  a sense  that  they  had  gained  an 
understanding  of  the  student’s  situation,  internalized  it,  and  were  concerned  for  his  future.  They  had  been  able 
to  incorporate  material  about  the  student’s  current  level  of  performance  and  made  inferences  about  what  may 
happen  to  him  in  his  future.  They  als  o wanted  to  make  sure  they  are  responsive  to  his  perceived  needs  (“. . .if 
Andres  needed  us  to.”)  and  want  to  be  able  to  address  his  needs  in  their  planning. 

The  emotional  connection  to  the  characters  was  particularly  evident  when  they  expressed  frustration 
about  Andres’  general  education  teacher.  Typical  participant  remarks  include  strongly  worded,  judgmental 
comments: 

What  bothers  me  most  about  his  situation  is  his  general  ed  [sic]  teacher.  I don't  feel  as  though  she  puts 
100%  effort  towards  Andres.  I understand  that  she  has  other  students  in  the  classroom,  but  she  doesn 't 
even  try  to  communicate  with  his  parents.  The  whole  situation  with  her  really  bothers  me. 

One  of  my  main  concerns  with  Andres  is  that  he  is  way  behind  the  rest  of  the  students  in  his  academics. 
If  he  doesn 't  catch  up,  he  is  just  going  to  get  further  and  further  behind. 

2.B)  PBL  challenges  students  to  resolve  ill-structured  problems 

One  of  the  key  characteristics  of  the  PBL  process  is  working  with  an  ‘ill-structured’  problem  that 
reflects  the  messy,  real-life  complications  of  problem-solving  (Howard,  2002).  Students  are  expected  to  solve 
problems  without  enough  information.  They  seek  out  information  from  various  sources,  prioritize  relevant 
information  and  filter  out  irrelevant  information  in  order  to  define  the  problem.  After  defining  the  problem, 
students  make  decisions  without  perfect  knowledge  and  are  not  sure  whether  the  decision  is  the  correct  one 
(Duffy  & Cunningham,  2001). 

The  MUSE  case  provided  comprehensive  albeit  incomplete  information  about  the  student.  Participants 
were  able  to  review  work  samples,  video  footage  of  interactions  in  the  classroom  and  recess,  and  comments 
from  teachers,  a parent  representative,  and  other  school  personnel.  Despite  the  variety  of  information,  a number 
of  the  teams  expressed  frustration  at  the  perceived  lack  of  information.  For  example: 

I also  feel  that  we  don 't  have  enough  info  to  really  make  a realistic  IEP. 

It  would  have  been  a better  activity  if  we  had  more  information  about  Andres  b/c  [sic]  I felt  for  the 
most  part  we  were  really  struggling  to  make  a decision  because  of  lack  of  information. 

Despite  concerns  about  the  lack  of  information,  participants  were  required  to  utilize  the  resources  and 
information  provided  and  accomplish  the  activity  goals  by  engaging  in  collaborative  learning  (Duffy  & 
Cunningham,  2001).  It  was  necessary  for  the  teams  to  develop  a process  to  use  the  information  they  had  and 
overcome  their  perceptions  that  it  was  an  inadequate  amount  from  which  to  make  a decision.  A number  of 
participants  shared  their  teams’  problem-solving  strategies  with  an  ‘ill-structured’  (Jonassen  & Hernandez- 
Serrano,  2002)  scenario  in  response  to  a final  reflective  question  that  asked  participants  to  identify  the 
challenges  their  group  experienced  and  describe  how  they  resolved  them: 

Our  biggest  challenge  in  the  group  was  that  we  really  did  not  know  Andres.  We  had 
to  make  assumptions  about  what  was  going  on  and  how  to  assess  him  and  it  was 
hard  to  make  decisions.  We  really  never  resolved  it,  but  did  the  best  that  we  could 
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with  the  information  that  we  were  given. 

We  did  not  feel  like  we  had  all  the  information  we  wanted  to  complete  the  activity. 

We  overcame  it  by  talking  it  out  with  each  other  and  using  everyone's  input  to  try 
and  complete  the  information  as  much  as  possible. ..or  at  least  make  up  a scenario 
that  was  as  close  as  we  could  get  to  the  actual  truth.  There  was  also  several  times  we 
just  disagreed  on  certain  things,  but  we  would  talk  through  them  as  well.  I think  our 
team  did  an  excellent  job  talking  through  things  and  reaching  conclusions  based  on 
our  knowledge. 

All  teams  developed  an  educational  plan  for  the  student  and  were  able  to  make  a final  decision  about 
whether  or  not  they  would  recommend  that  the  student  be  referred  to  special  education.  This  ability  for 
participants  to  accomplish  the  goal  indicates  that  the  PBL  module  was  appropriately  ‘ill -structured’.  Students 
perceived  they  didn’t  have  all  the  answers,  but  were  able  to  successfully  collaborate  and  problem-solve 
sufficiently  enough  to  respond  to  the  module  questions. 

This  incomplete  problem  structure  mirrors  how  a real  IEP  meeting  may  progress  because  each  team 
member  brings  to  the  meeting  their  personal  interactions  and  experiences  with  the  student.  Some  team 
members  may  have  ownership  of  certain  information  as  well  (Gerber  et  ah,  1999).  For  example,  a teacher 
would  have  special  insight  on  classroom  management,  whereas  a school  psychologist  may  have  one-on-one 
counseling  information  about  the  student. 

Question  3 ) How  does  the  use  of  multimedia  in  the  module  impact  satisfaction  with  the  PBL  experience? 

Participant  responses  about  their  perceptions  of  the  impact  of  media  (video,  audio,  images)  supported 
the  survey  findings.  The  interviewee  was  asked  her  opinion  on  the  impact  of  providing  the  MUSE  module 
through  a text  -based  case  study  without  multimedia.  The  student  indicated  that  it  would  have  been  a different 
experience  for  them: 

I don’t  think  it  would  have  been  as  interactive.  I wouldn’t  have  been  as  responsive  to  it,  I 
don ’t  think.  If  they  just  give  you  paper,  it  would  not  have  been  as  much  fun. 

In  addition,  it  increased  her  later  recollection  of  the  Andres  case.  One  student  indicated  that  media 
increased  the  connection  she  felt  to  the  student  and  helped  personalize  the  context.  When  asked  if  the  she  felt 
attached  to  the  Andres  case,  the  student  offered: 

I would  not  say  an  attachment,  but  it  was  more  like..  I could  picture  him.  I could  picture  kinda 
[sic]  what  he  was  thinking.  It  was  easier  that  way.  I could  picture  him  in  my  head.  I could 
know  what  he  was  like  and  see  him.  Emotionally  attached  no,  it  definitely  made  him  more 
interactive  knowing  what  he  was  like. 

Through  a variety  of  qualitative  and  quantitative  measures,  students  were  able  to  give  insight  into  their 
interactions  and  reactions  to  this  case-based  multimedia  PBL  module.  Their  comments  indicated  that  while 
they  grappled  with  a perceived  lack  of  information,  through  the  process  of  forming  a solution,  they  collectively 
increased  their  knowledge  of  special  educational  processes,  developed  collaboration  skills,  and  began  to 
develop  a connection  with  their  future  professional  community.  One  student  summarized  it  best  when  she  said: 

I really  enjoyed  this  activity.  I feel  that  this  will  soon  affect  me  as  I become  a teacher 
in  a year.  I think  this  type  of  teaching  (PBL)  is  very  important  in  teaching  education. 

This  is  the  sort  of  stuff  we  will  be  involved  with  and  this  will  help  us  become  more 
knowledgeable  about  this  subject. 


Conclusion 

The  results  described  by  this  study  offer  insight  into  the  effectiveness  of  using  multimedia  PBL 
strategies  to  teach  special  education  concepts  to  pre -service  elementary  education  teachers.  Student  reaction 
indicates  that  the  module  provided  an  effective  learning  experience.  In  the  future,  follow  up  with  students  once 
they  are  educators  would  provide  an  additional  measure  of  the  impact  of  this  case  on  their  practice. 
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Abstract 

Animation  is  being  used  extensively  for  instructional  purposes;  however,  it  has  not  been  found  to  be 
effective  on  measures  of  higher  order  learning  (concepts,  rules,  procedures)  within  the  knowledge  acquisition 
and  knowledge  integration  domains.  The  purpose  of  this  study  was  to  examine  the  instructional  effectiveness  of 
two  visual  scaffolding  strategies  (simple  and  complex  scaffolding)  used  to  complement  animated  instruction. 
About  90  undergraduate  level  students  were  randomly  assigned  to  three  treatments  (control,  simple  and 
complex).  After  receiving  their  respective  instructional  presentation  students  took  four  tests  - drawing, 
identification,  terminology  and  comprehension.  The  results  of  a preliminary  study  indicated  that  animation  has 
a significant  impact  on  acquisition  of  factual  and  conceptual  knowledge.  On  the  other  hand,  visual  scaffolding 
strategies,  used  as  a complement  to  instruction  that  already  involved  animation  did  not  have  a significant 
impact  on  students  ’ performance  on  measures  of  higher  order  learning. 

Introduction  and  Theoretical  Framework 

Animation  in  Multimedia  Instruction 

(Mayer  & Moreno,  2002)  have  defined  multimedia  instructional  environments  as  ones  in  which 
“learners  are  exposed  to  material  in  verbal  (such  as  on  screen  text  or  narration)  as  well  as  pictorial  form 
(including  static  materials  such  as  photos  or  illustration,  and  dynamic  material  such  as  video  or  animation)”  (pg. 
87).  The  authors  propose  two  theories  of  how  students  learn  from  words  and  pictures: 

1.  Information  Delivery  Theory  of  Multimedia  Learning:  Based  on  the  theory  that  learning  involves  adding 
information  to  one's  memory  (Mayer,  1996).  Multimedia  instruction  is  effective  in  delivering  information 
effectively  to  both  types  of  learners  - learners  that  prefer  verbal  presentation  and  those  that  prefer  pictorial 
presentations. 

2.  Cognitive  Theory  of  Multimedia  Learning:  Meaningful  learning  occurs  when  students  mentally  construct 
coherent  knowledge  representations  (Mayer,  1996).  This  theory  is  based  on  three  assumptions:  (1)  humans  have 
separate  channels  of  processing  visual  and  verbal  representation  (dual-channel  theory)  (2)  the  capacity  of  short 
term  memory  is  limited,  and  (3)  meaningful  learning  (knowledge  integrations)  occurs  when  learners  actively 
engage  in  cognitive  processes  such  as  selecting,  organizing  and  representing  knowledge. 

Further,  (Mayer  & Moreno,  2002)  have  examined  the  role  of  animation  in  multimedia  learning 
environments.  Animation  is  defined  as  “[Animation]  refers  to  a simulated  motion  picture  depicting  movement 
of  drawn  (or  simulated)  objects”  (pg.  88).  They  found  that  “Animation  can  promote  learner  understanding  when 
used  in  ways  that  are  consistent  with  the  cognitive  theory  of  multimedia  learning.” 

Potential  Problems  with  Multimedia  Instruction  using  Animation 

Animation  and  simulations  are  being  utilized  at  all  levels  of  instruction.  However,  most  research 
which  has  identified  positive  gain  from  animation  has  reported  it  at  the  fact  and  concept  levels  (Reiber,  1990; 
Dwyer,  2003).  One  hypothesis  that  may  be  proposed  to  explain  this  phenomenon  is  that  when  students  are 
expected  to  learn  a hierarchy  of  learning  outcomes,  the  cognitive  load  associated  with  the  animated  presentation 
and  the  content  complexity  provides  a stimulus  field  which  is  too  complex  for  effective  assimilation  (Young, 
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1993).  Another  hypothesis  is  that  “The  ineffectiveness  of  animation  in  facilitating  higher  level  cognitive 
functions  may  be  because  learners  do  not  possess  the  prerequisite  facts  and  concepts  to  use  in  constructing  rules 
and  principles  necessary  for  higher  order  comprehension.”  (Dwyer,  2003) 


Scaffolding  in  Instruction 

“[ A scaffold]  lends  consciousness  to  a child  who  does  not  have  on  his  own  ” (Bruner,  1986,  p.  86) 

Scaffolding  has  been  defined  as  a strategy  which  involves  supporting  learners  by  limiting  the 
complexities  of  the  learning  content.  In  her  paper  (Dabbagh,  2003)  cites  definitions  of  scaffolding  provided  in 
(Young,  1993)-  “Scaffolding  involves  supporting  novice  learners  by  limiting  the  complexities  of  the  learning 
context  and  gradually  removing  those  limits  (a  concept  known  as  fading)  as  learners  gain  the  knowledge,  skills 
and  confidence  to  cope  with  the  full  conplexity  of  the  context”;  and  (Jarvela,  1995;  Pressley  & et  ah,  1996)  - 
“Assistance  to  learners  is  provided  on  an  as  -needed  basis  and  as  their  task  competence  increases,  fading  of 
assistance  is  gradually  administered  to  allow  learners  to  complete  the  task  independently.” 

(Stone,  1998)  defines  scaffolding  as  a metaphor  for  the  process  by  which  adults  or  more 
knowledgeable  peers  guide  children's  learning  and  development.  According  to  (Stone,  1998),  “In  providing 
temporary  assistance  to  children  as  they  strive  to  accomplish  a task  just  out  of  their  competency,  adults  are  said 
to  be  providing  a scaffold,  much  like  that  used  by  builders  in  erecting  a building”  (p.  344).  (Wood,  Bruner,  & 
Ross,  1976)  [as  cited  in  (Stone,  1998)]  describe  scaffolding  as  a form  of  adult  assistance  “that  enables  a child  or 
novice  to  solve  a problem,  carry  out  a task  or  achieve  a goal  which  would  be  beyond  his  unassis  ted  efforts”  (p. 
90). 

(Wood  et  ah,  1976)  [as  cited  in  (Stone,  1998)]  identified  six  types  of  assistance  that  an  adult  tutor 
could  provide  to  scaffold  learning:  (a)  recruitment  of  children’s  interest,  (b)  reduction  in  degrees  of  freedom,  (c) 
maintaining  goal  orientation,  (d)  highlighting  critical  task  features,  (e)  controlling  frustration,  and  (f) 
demonstrating  idealized  solution  paths.  (Stone,  1998)  in  referring  to  (Wood  et  ah,  1976)  writes  “It  is  important 
to  note  that  this  list  includes  (a)  perceptual  components  (e.g.  highlighting  critical  task  features),  (b)  cognitive 
components  (e.g.  reduction  in  degrees  of  freedom),  and  (c)  affective  components  (e.g.  controlling  frustration).” 

In  her  paper  (Butler,  1998)  describes  how  the  scaffolding  metaphor  is  used  in  Strategic  Content 
Learning,  an  instructional  approach  that  promote  strategic  learning  in  students  with  learning  disabilities.  She 
writes,  “The  scaffolding  metaphor  has  made  significant  contributions  to  our  understanding  of  the  characteristics 
of  effective  instruction.  Those  contributions  include  an  emphasis  on  important  instructional  characteristics:  (a) 
support  should  be  flexibly  calibrated  to  meet  students’  needs;  (b)  support  should  be  either  increased  or  faded 
depending  on  how  independently  students  regulate  their  learning;  (c)  support  should  be  provided  in  the  context 
of  a meaningful  task;  (d)  support  is  best  provided  by  means  of  interactive  dialogues  conducted  during 
collaborative  problem  solving;  and  (e)  rather  than  breaking  tasks  into  subskills,  support  should  be  provided  for 
subskills  as  they  occur  in  the  context  of  meaningful  tasks.” 

(Hannafin,  Hannafin,  Land,  & Oliver,  1997)  have  proposed  a model  of  scaffolding  in  open-ended 
learning  environments.  They  delineate  four  categories  of  scaffolds:  (a)  Conceptual  scaffolding  (helps  students 
determine  what  to  consider  when  solving  a problem),  (b)  Metacognitive  scaffolding  (supports  the  underlying 
processes  associated  with  individual  learning  management)  (c)  Procedural  scaffolding  (helps  learners  by 
providing  hints  on  how  to  utilize  available  resources  and  tools),  and  (d)  Strategic  scaffolding  (provides  support 
for  how  to  utilize  strategies). 

Based  on  the  1976  article  by  (Wood  et  ah,  1976),  (Pea  & Mills,  2004)  has  described  the  processes  by 
which  scaffolding  is  “functioned”  for  the  learner:  “1.  Channeling  and  focusing:  Reducing  the  degrees  of 
freedom  for  the  task  at  hand  by  providing  constraints  that  increase  the  likelihood  of  the  learner’s  effective 
action;  recruiting  and  focusing  attention  of  the  learner  by  marking  relevant  task  features  (in  what  is  otherwise  a 
complex  stimulus  field),  with  the  result  of  maintaining  directedness  of  the  learner’s  activity  toward  task 
achievement.  2.  Modeling:  Modeling  more  advanced  solutions  to  the  task."  (pg.  432). 

Origins  of  the  Scaffolding  Metaphor 

The  origins  of  the  scaffolding  metaphor  lie  in  the  social  constructivist  theoretical  tradition.  Scaffolding 
has  clear  connections  with  Vygotsky’s  idea  of  Zone  of  Proximal  Development.  According  to  (Bull  et  al.,  1999), 
“When  in  the  zone  of  proximal  development  for  a particular  skill  or  a piece  of  information,  a learner  is  ready  to 
learn  but  lacks  certain  prerequisites.  Scaffolding  is  an  interactive  process  in  which  a teacher  or  facilitator  assists 
such  a learner  to  build  a ‘structure’  to  contain  and  frame  the  new  information”  (p.  240). 

(Stone,  1998)  has  pointed  out  that  “Although  the  initial  use  of  the  scaffolding  metaphor  was  largely 
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pragmatic  and  atheoretical,  in  subsequent  discussion  it  was  increasingly  linked  with  Vygotsky’s  (1962,  1978) 
developmental  theory.  ...The  implicit  link  between  Vygotsky’s  ZPD  and  the  scaffolding  metaphor  was  first 
made  explicit  by  Cazden  (1979)”  (p.  345). 

Scaffolding  and  Cognitive  Theory 

Cognitive  theory  looks  at  understanding  as  being  determined  by  the  previous  experiences  of  the 
learner,  his  past  knowledge  and  the  ways  in  which  this  information  has  been  stored  (memory  structures 
determine  how  new  information  will  be  assimilated  or  represented).  (Bull  et  ah,  1999)  have  related 
understanding  with  scaffolding  as  such  ‘‘To  be  able  to  learn  from  particular  information,  a learner  must  have 
sufficient  background  knowledge  to  be  able,  with  help,  to  start  to  process  the  new  information  into  personal 
knowledge.  . . .When  scaffolding  is  necessary,  the  teacher  should  try  to  minimize  the  cognitive  load  by  setting 
the  environment  conditions  so  that  the  student  can  both  recall  and  use  information  that  he/she  already  knows  to 
perform  most  of  the  task  (tie  the  new  material  to  the  old).  Therefore  the  student  has  only  to  learn  a limited 
amount  of  new  information  to  be  successful”  (p.  242). 

Visual  scaffolding 

Much  research  has  been  done  on  the  role  of  pictures  in  text.  Pictures  can  help  learning  by  establishing 
a setting,  contributing  to  text's  coherence  and  reinforcing  the  text.  (Levin  & Mayer,  1993)  have  proposed  seven 
“C”  principles  for  explaining  why  pictures  facilitate  learning  - pictures  improve  student  learning  from  text  by 
making  it  text  more  concentrated,  compact/concise,  concrete,  coherent,  comprehensible,  correspondent,  and 
codable. 

(Cuevas,  Fiore,  & Oser,  2002)  have  studied  how  instructional  strategies  (such  as  use  of  diagrams  in 
instruction)  in  complex  task  training  environments  can  be  used  to  scaffold  learners’  cognitive  and  metacognitive 
processes,  especially  for  low  ability  learners.  Their  findings  suggest  that  incorporating  diagrams  into  training 
facilitated  performance  on  measures  of  integrative  knowledge  (they  found  no  significant  effect  on  measures  of 
declarative  knowledge).  They  write  ‘‘Diagrams  additionally  facilitated  the  development  of  accurate  mental 
models  and  significantly  improved  the  instructional  efficiency  of  the  training.  Finally  diagrams  effectively 
scaffold  participants’  metacognition,  improving  their  metacomprehension  accuracy  (i.e.  their  ability  to  actually 
monitor  their  comprehension)”  (p.  433).  ‘‘There  are  several  theories  that  elucidate  why  inclusion  of  illustrations, 
such  as  pictures  and  diagrams  leads  to  better  understanding  of  the  presented  material  and  improved  retention 
and  application  of  its  concepts.  One  theory  suggests  that  diagrams  repeat  the  information  in  the  text.  . . .Another 
interpretation  of  positive  effects  of  diagrams  attributes  improved  learning  to  dual  coding  of  the  information  in 
memory.  Paivio  (1971)  proposed  that  verbal  and  nonverbal  (i.e.,  visual/  spatial)  information  are  processed  in 
separate,  functionally  distinct,  although  interconnected,  long  term  memory  systems.  ...Accordingly,  presented 
information  using  both  texts  and  diagrams  activates  more  than  one  mechanism  of  memory....  Therefore,  since 
the  information  is  processed  by  two  distinct  mechanisms,  encoding  is  reinforced,  and  retrieval  from  memory 
should  be  facilitated”  (p.  434). 

According  to  (Cuevas  et  ah,  2002)  diagrams  increase  the  efficiency  of  the  learner’s  information 
processing  by  decreasing  the  cognitive  load,  “Well-designed  instructional  programs  would  be  expected  to 
increase  the  efficiency  of  the  learner’s  information  processing,  so  that  fewer  cognitive  resources  are  required  for 
task  performance  after  training  (Paas  & Van  Merrienboer,  1993).  Within  the  context  of  the  mental  model 
approach  we  propose  that  diagrams  may  reduce  the  cognitive  load  on  working  memory  and  attention  associated 
with  complex  tasks  by  making  structural  relations  clearer  and  more  transparent  (Marcus  et  ah,  1996).  Thus, 
incorporating  diagrams  into  the  training  would  be  expected  to  result  in  higher  instructional  efficiency  (i.e., 
higher  performance  will  be  achieved  with  less  mental  effort  exerted.)”  (p.  437). 

Visual  Scaffolding  and  Cognitive  Theory 

(Cuevas  et  ah,  2002)  have  suggested  the  Metal  model  theory  as  a theory  for  why  diagrams  are  so 
effective  in  instruction.  In  the  Metal  model  theory  thinking  is  considered  equivalent  to  manipulating  internal 
representations  stored  in  the  mind.  According  to  (Cuevas  et  ah,  2002),  diagrams  may  serve  to  scaffold  the 
development  of  mental  models. 

Research  Hypothesis  and  Problem  Statement 


Research  Hypothesis 

It  is  hypothesized  that  visual  scaffolding  used  to  complement  animated  sequences  would  serve  to 
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emphasize  the  critical  attributes  to  be  learned,  thereby  reducing  the  cognitive  load.  This  will,  in  turn,  enable 
students  to  process  information  more  effectively.  The  visual  scaffolds  designed  to  complement  animation 
would  function  to  facilitate  generative  and  metacognitive  processes  necessary  to  facilitate  the  comprehension  of 
higher  level  learning  objectives  and  the  transfer  of  information  from  short  term  into  long  term  memory.  This 
notion  of  using  scaffolding  to  provide  procedural  guidance  to  more  effectively  process  the  information 
acquisition  has  been  supported  by  McLoughlin  & Oliver  (1999). 

Problem  Statement 

Dabbagh  (2003,  p.42)  has  hypothesized  that  “low  and  high  scaffolding  are  highly  correlated  with  the 
type  of  instructional  strategies  implemented  in  a learning  environment.”  The  purpose  of  this  study  was  to 
examine  the  instructional  effects  of  scaffolding  in  facilitating  higher  level  performance  outcomes.  Specifically, 
the  focus  of  this  study  was  to  examine  the  degree  to  which  two  levels  of  visual  scaffolding  strategies  (simple 
and  complex),  used  to  complement  animated  instruction,  facilitated  achievement  of  higher  level  performance 
outcomes  as  measured  by  four  criterion  tests. 

Instructional  Content  and  Dependent  Measures 


Instructional  Content 

The  instructional  content  used  in  the  study  is  related  to  the  physiology  and  functions  of  the  human 
heart.  This  content  was  selected  because  it  provided  a hierarchy  of  learning  objectives  (from  facts  to  problem 
solving).  Problem  solving  required  learning  the  terminology  of  the  human  heart,  location  of  the  parts  and  their 
respective  functions,  and  positions  during  the  systolic  and  diastolic  phases.  The  dependent  variables  in  the  study 
were  achievement  on  test  measuring  different  levels  of  learning.  Achievement  was  measured  in  terms  of  facts, 
concepts,  rules/procedures  and  comprehension.  A 20-item  test  was  developed  for  each  of  these  criterion 
measures.  Average  Kuder-Richardson  Formula-20  reliability  coefficients  from  a random  sampling  of  studies 
(Dwyer,  1978)  are:  .83  for  the  Terminology  Test,  .81  for  the  Identification  Test,  .83  for  the  Drawing  Test,  .77 
for  the  Comprehension  Test,  and  .92  for  the  Total  Test.  Following  are  descriptions  of  the  criterion  measures 
employed,  (Ibid.  45-47). 

Dependent  Measures 

Drawing  Test.  The  objective  of  the  drawing  test  was  to  evaluate  student  ability  to  construct  and/or 
reproduce  items  in  their  appropriate  context.  The  drawing  test  provided  the  students  with  a 
numbered  list  of  terms  corresponding  to  the  parts  of  the  heart  discussed  in  the  instructional 
presentation.  The  students  were  required  to  draw  a representative  diagram  of  the  heart  and  place  the 
numbers  of  the  listed  parts  in  their  respective  positions.  For  this  test  the  emphasis  was  on  the  correct 
positioning  of  the  verbal  symbols  with  respect  to  one  another  and  in  respect  to  their  concrete 
referents. 

Identification  Test.  The  objective  of  the  identification  test  was  to  evaluate  student  ability  to  identify 
parts  or  positions  of  an  object.  This  multiple -choice  test  required  students  to  identify  the  numbered 
parts  on  a detailed  drawing  of  a heart.  Each  part  of  the  heart,  which  had  been  discussed  in  the 
presentation,  was  numbered  on  a drawing.  The  objective  of  this  test  was  to  measure  the  ability  of  the 
student  to  use  visual  cues  to  discriminate  one  structure  of  the  heart  from  another  and  to  associate 
specific  parts  of  the  heart  with  their  proper  names. 

Terminology  Test.  This  test  consisted  of  items  designed  to  measure  knowledge  of  specific  facts, 
terms,  and  definitions.  The  objectives  measured  by  this  type  of  test  are  appropriate  to  all  content  areas 
which  have  an  understanding  of  the  basic  elements  as  a prerequisite  to  the  learning  of  concepts,  rules, 
and  principles. 

Comprehension  Test.  Given  the  location  of  certain  parts  of  the  heart  at  a particular  moment  of  its 
functioning,  the  student  was  asked  to  determine  the  position  of  other  specified  parts  or  positions  of 
other  specified  parts  of  the  heart  at  the  same  time.  This  test  required  that  the  students  have  a thorough 
understanding  of  the  heart,  its  parts,  its  internal  functioning,  and  the  simultaneous  processes  occurring 
during  the  systolic  and  diastolic  phases.  The  comprehension  test  was  designed  to  measure  a type  of 
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understanding  in  which  the  individual  can  use  the  information  being  received  to  explain  some  other 
phenomenon. 

Total  Test  Score.  The  items  contained  in  the  individual  criterion  tests  were  combined  into  a 
composite  test  score.  The  purpose  was  to  measure  total  achievement  of  the  objectives  presented  in  the 
instructional  unit. 


Research  Methodology 

Two  pilot  studies  were  conducted  to  develop  the  instruction  that  is  used  as  the  control  treatment  in  this 

study. 

Pilot  Study  1 

In  the  first  pilot  study,  programmed  instruction  focusing  on  facts  and  concepts  necessary  for  higher 
order  learning  was  prepared  and  tested.  The  rationale  for  developing  the  programmed  instruction  was  a 
hypothesis  from  (Dwyer,  2003):  “The  ineffectiveness  of  animation  in  facilitating  higher  level  cognitive 
functions  may  be  because  learners  do  not  possess  the  prerequisite  facts  and  concepts  to  use  in  constructing  rules 
and  principles  necessary  for  higher  order  comprehension.”  Students  were  presented  with  this  instruction 
followed  by  the  heart  content  tests.  Their  scores  on  the  drawing  and  identification  test  (facts  and  concepts)  were 
found  to  be  significantly  better  than  those  students  that  took  regular  instruction  (in  previous  studies).  Students’ 
scores  on  the  terminology  and  comprehension  tests  (rules  and  procedures)  were  still  low.  These  results  showed 
that  although  programmed  instruction  was  effective  in  transfer  of  facts  and  concepts;  rules  and  procedures  still 
needed  attention. 

Pilot  Study  2 

Based  on  item  analysis  of  the  identification  and  comprehension  test  (rules  and  procedures),  points  in 
the  instruction  that  needed  improvement  were  identified.  Animation  (developed  using  Macromedia  Flash)  was 
designed  and  placed  at  these  points.  A second  pilot  study  with  138  students  and  three  treatments:  Condition  A 
(Control  group:  regular  instruction).  Condition  B (programmed  instruction),  and  Condition  C (programmed 
instruction  + animation)  was  conducted.  Results  from  this  study  again  showed  that  the  programmed  instruction 
was  effective  in  transferring  facts  and  concepts.  It  should  be  noted  that  no  significant  gains  in  the  terminology 
and  comprehension  tests  (rules  and  procedures)  were  obtained  in  Condition  C.  (Table  la  and  lb) 

An  item  analysis  of  the  identification  and  comprehension  test  for  Condition  C in  the  second  pilot  study 
was  conducted  to  identify  points  in  the  instruction  that  needed  further  improvement.  Simple  and  complex 
scaffolding  was  designed  and  placed  at  these  points. 

Development  of  Simple  and  Complex  Scaffolding  Treatment 

We  have  used  the  cognitive  model  to  define  simple  and  complex  scaffolding  as  such  - simple 
scaffolding  instigates  lower  levels  of  cognitive  processing  in  learners  as  compared  to  complex  scaffolding, 
which  instigates  higher  levels  of  cognitive  processing  in  the  learner. 

Another  dimension  along  which  scaffolding  can  be  differentiated  is  suggested  by  (Azevado  et  al, 

2004).  (Azevado  et  al,  2004)  examined  the  role  of  different  scaffolding  interventions  in  facilitating  students’ 
shift  in  mental  models.  They  found  adaptive  scaffolding  (access  to  a tutor  and  specific  goals)  facilitated  shift  in 
a learner’s  mental  models  significantly  more  than  fixed  scaffolding  (access  only  to  specific  goals). 

Note:  It  has  been  the  experience  of  the  author  that  instruction  that  uses  complex  scaffolding  is  more 
challenging  from  an  instructional  design  and  development  point  of  view.  For  example  simple  scaffolding  for 
this  study  is  designed  using  simple  HTML  forms,  whereas  the  complex  scaffolding  is  designed  with  Java 
applets. 

Visual  Scaffolding  using  Transformational  (Mnemonic)  Function  of  Images 

The  use  of  diagrams  as  mnemonics  to  provide  scaffolding  is  supported  by  research.  (Levin,  1981)  has 
delineated  five  functions  that  pictures  serve  in  text  processing:  decorational,  representational,  organizational, 
interpretational  and  transformational.  According  to  (Carney  & Levin,  2002)  decorational  pictures  “simply 
decorate  the  page,  bearing  little  or  no  relationship  to  the  text  content”;  representational  pictures  “mirror  part  or 
all  of  the  text  content  and  are  by  far  the  most  commonly  used  type  of  illustration”;  organizational  pictures 
“provide  a useful  structural  framework  for  the  text  content”;  interpretational  pictures  “help  to  clarify  difficult 
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text”;  and  transformational  pictures  “include  systematic  mnemonic  (memory  enhancing)  components  that  are 
designed  to  improve  a reader’s  recall  of  text  information”.  It  is  worth  noting  that  in  theory  research  (Carney  & 
Levin,  2002)found  that  “Purely  decorational  pictures  exhibited  virtually  no  beneficial  text -learning  effects, 
whereas  the  remaining  effect  sizes  ranged  from  moderate  benefits  (for  representational  pictures)  to  quite 
substantial  benefits  (for  transformational  pictures)”  (pg.  7-8). 

Moreover,  on  the  basis  of  (Levin  et  al,  1990),  (Carney  & Levin,  2002)  suggest  the  following  about  the 
interpretational  function  of  pictures  in  instruction,  “Of  particular  interest  to  these  investigators  was  whether 
mnemonic  illustrations  could  enable  students  go  beyond  the  information  given  and  assist  them  in  performing 
higher  order  cognitive  application  tasks  such  as  those  involving  inference,  problem  solving,  and  analogical  and 
syllogistic  reasoning  based  on  the  botany  content”  (pg.  18).  Further  “Combined  with  separate  mnemonic 
illustrations  for  solidifying  unfamiliar  terminology  and  definitions,  the  pictorial  mnemonomy  was  found  to  be  a 
potent  facilitator  of  students’  information  reconstruction  and  application  performance  both  on  immediate  tests 
and  on  delayed  tests  up  to  2 months  later”  (pg.  18).  Finally,  according  to  (Carney  & Levin,  2002),  (Atkinson  et 
al.,  1999) have  argued  that  “that  the  ready  access  to  information  that  mnemonic  strategies  afford  can  facilitate 
students’  acquisition  of  higher  order  concepts  and  skills”  (pg.  20). 

Results 

For  the  present  study,  there  conditions  were  used:  Condition  A (programmed  instruction  + animation), 
Condition  B (programmed  instruction  + animation  + simple  scaffolding),  and  Condition  C (programmed 
instruction  + animation  + complex  scaffolding).  Dependent  measures  were  scores  achieved  on  the  drawing, 
identification,  terminology,  and  comprehension  tests  . 87  students  were  randomly  assigned  to  the  three  treatment 
groups.  Data  from  each  criterion  measure  was  analyzed  collectively  and  individually  to  comprehensively 
examine  the  contributions  of  visual  scaffolding  in  complementing  animation.  No  significant  differences  in 
scores  were  found.  (Table  2a  and  2b) 


Discussion 

According  to  (Mayer  & Moreno,  2003),  a potential  problem  of  multimedia  learning  environments  is 
that  processing  demands  evoked  by  the  learning  task  (words  and  pictures)  may  exceed  the  processing  capacity 
of  cognitive  systems.  Such  a situation  is  called  cognitive  overload.  (Mayer  et  al,  2003)  describe  three  kinds  of 
cognitive  demands:  (1)  essential  processing  (cognitive  processes  that  are  required  to  make  sense  of  the 
presented  material);  (2)  incidental  processing  (cognitive  processes  that  are  due  to  the  design  of  the  learning 
task);  and  (3)  representational  holding  (cognitive  resources  used  to  hold  a metal  model  in  working  memory). 
Cognitive  overload  occurs  when  the  sum  of  these  processing  demands  exceeds  the  processing  capacity  of  the 
learner’s  cognitive  system. 

The  insignificant  results  of  this  study  may  be  explained  by  the  increase  in  cognitive  load  that  visual 
scaffolding  and  animation  put  on  the  learners.  In  other  words,  positive  effect  of  visual  scaffolding  and 
animation  may  be  cancelled  by  an  increase  in  task  complexity. 

Conclusions 

The  results  of  the  analyses  indicated  that  specific  types  of  visual  scaffolding  (simple  and  complex)  are 
important  variables  for  facilitating  specific  types  of  performance  outcomes.  Initial  interpretation  of  the  results 
indicated  that  visual  scaffolding  strategies,  specifically  designed,  developed  and  positioned,  have  the  potential 
for  focusing  and  illustrating  procedural  understanding  thereby  reducing  the  cognitive  load  associated  with  the 
higher  processing  levels  in  the  knowledge  acquisition  domain. 
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Motivational  Influences  in  Self-Directed  Online  Learning  Environments: 

A Qualitative  Case  Study 

Kyong-Jee  Kim 
Indiana  University 

Introduction 

The  Web  technology  is  changing  the  way  people  learn,  work,  and  socialize  (Bonk  & King,  1998). 

More  and  more  people  are  turning  to  the  Web  technology  for  their  learning  needs  due  to  the  flexible  delivery 
system  of  the  Web.  Although  the  effectiveness  of  Web-based  instruction  has  been  proven  in  many  studies  (Jung 
& Rha,  2000),  high  learner  drop-out  rates  have  been  a concern  in  Web-based  instruction,  which  have  also  been 
the  case  in  distance  education  and  computer-based  instruction  (Diaz,  2002;  Islam,  2002;  Moore  & Kearsley, 
1996). 

Past  studies  on  the  factors  of  learner  attrition  in  distance  education  suggest  that  lack  of  time  and  lack  of 
motivation  are  the  major  causes  of  the  problem  (Bonk,  2002;  Gibson,  1998;  Visser,  Plomp,  Amirault,  & Kuiper, 
2002;  Wolcott  & Burnham,  1991;  Zvacek,  1991).  Although  instructional  designers  or  instructors  do  not  have 
control  over  the  learner’s  time,  they  can  have  some  influence  over  learner  motivation  as  it  tends  to  change 
through  instruction  (Coldeway,  1991;  Song  & Keller,  1999).  Therefore,  attention  needs  to  be  paid  to  improving 
learner  motivation  to  address  the  issue  of  learner  attrition  in  Web-based  instruction. 

Research  abounds  about  the  importance  of  learner  motivation  in  learning.  Past  studies  have 
consistently  reported  that  motivation  makes  a significant  impact  on  the  student’s  achievement.  In  addition, 
successful  learning  experiences,  which  affect  continuing  motivation  of  the  learner,  are  conducive  to  life-long 
learning  (Wlodkowski,  1993),  which  is  critical  for  adults  as  society  becomes  more  complex  and  changes  rapidly 
more  than  ever.  As  with  traditional  instruction,  learner  motivation  is  an  important  instructional  design 
component  of  Web-based  instruction  (Bonk,  2002;  Ritchie  & Hoffman,  1997).  Although  the  importance  of 
learner  motivation  for  Web-based  instruction  has  been  recognized,  there  is  a lack  of  research  on  theories  and 
practices  of  the  design  of  motivating  Web-based  instruction  (Keller,  1999;  Song,  2000). 

Motivation  is  critical  for  the  success  of  online  learners.  E-learning  is  a rapidly  growing  market  and  is  expected 
to  be  so  in  the  future.  A recent  survey  reported  that  the  U.S.  e-learning  market  in  2002  was  $10.3  billion 
(Adkins,  2002).  It  is  projected  that  the  U.S.  e-learning  market  will  grow  to  $83.1  billion  in  2006.  Considering 
this  large  amount  of  spending  on  e-learning,  it  is  imperative  that  the  investment  to  be  worthwhile  for  the 
stakeholders.  To  accomplish  that  goal,  we  need  to  provide  online  learners  with  a learning  environment  that 
builds  success  for  their  learning.  Fostering  adequate  motivation  for  the  online  learner  is  one  of  the  critical 
factors  for  creating  a successful  online  learning  environment  (Hofmann,  2003). 

Yet,  responding  to  the  motivational  requirements  of  learners  in  self-directed  online  instruction,  which 
is  the  instructional  approach  in  most  online  computer  training  courses,  is  a great  challenge  due  to  the  lack  of 
interactions  in  such  learning  environments  (Bonk  & Dennen,  2003;  Cornell  & Martin,  1997;  Keller,  1999). 
Problems  resulting  in  symptoms  of  demotivation  may  also  stem  from  issues  other  than  motivation  - i.e.,  lack  of 
skills,  environmental  factors,  etc.  (Keller,  1999).  Therefore,  a systematic  approach  to  analyzing  the  problems  of 
learner  motivation  is  warranted  for  our  better  understanding  of  the  motivational  needs  of  learners  in  self-paced 
online  learning  environments. 

The  purpose  of  this  study  is  to  investigate  the  problems  associated  with  learner  motivation  in  Web- 
based  instruction,  in  particular  in  self-directed  online  learning  environments.  This  study  is  interested  in 
identifying  and  exploring  what  motivate  or  demotivate  learners  from  completing  a self-directed  online  course, 
which  have  implications  for  designing  motivating  online  learning  environments.  In  more  detail,  this  study  will 
answer  the  following  questions; 

• What  are  motivating  and  inhibiting  factors  to  learn  in  self-directed  online  learning  environments? 

• Does  learner  motivation  change  during  instruction?  if  so,  how? 

• Are  there  individual  differences  in  learners’  motivational  levels  in  self -directed  online  learning 

environments? 

The  results  of  this  study  are  expected  to  increase  our  understanding  of  the  motivational  needs  of  the 
participants  of  self-directed  online  computer  training  by  identifying  what  motivate  or  demotivate  them  to  learn 
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computer  skills  in  a self -directed  learning  environment.  The  results  of  the  study  are  expected  to  inform 
instructional  designers  of  how  to  design  a motivating  online  learning  environment. 

Background  of  the  Study 

his  study  investigated  learners  of  self-directed  online  courses  to  answer  aforementioned  questions. 
Here,  self-directed  online  courses  refer  to  courses  delivered  via  the  Web  in  which  learners  go  through 
instructional  materials  delivered  via  the  Web  at  their  own  pace  without  the  presence  of  an  instructor.  Adult 
learners  can  participate  in  online  learning  in  various  contexts,  yet  the  self-directed  online  learning  format  is  the 
focus  of  this  study  because  self-directed  online  learning  is  a primary  instructional  format  in  training  settings  for 
adult  learners  (Driscoll,  2002;  Galvin,  2002) . 

The  courses  that  the  study  participants  took  were  offered  by  a major  U.S.  e-learning  vendor,  who 
offers  over  3,000  online  courses  to  20  million  learners  per  year  worldwide.  Those  courses  are  offered  to  adult 
learners  in  various  educational  and  workplace  settings.  The  course  format  is  stand-alone,  typically  6-8  hours 
long,  self-paced  instruction  delivered  via  the  Web.  The  topics  covered  in  those  courses  include  desktop 
applications  (e.g.,  Microsoft  Office  products),  computer  programming  (e.g.,  JAVA,  Oracle,  MS  .NET),  soft 
skills  development  (e.g.,  coaching  skills,  consulting  skills),  and  special  topics  tailored  to  the  needs  of  specific 
organizations  or  fields. 

The  learners  participated  in  this  study  took  self-directed  online  courses  either  in  school  or  work 
settings.  The  learners  in  school  settings  took  the  online  courses  offered  by  the  university  either  for  personal 
development  or  as  assigned  by  their  course  instructois.  The  learners  in  work  settings  also  took  the  online 
courses  either  for  personal  development  or  to  improve  their  job  skills. 

Literature  Review 

Motivation  by  definition  is  the  degree  of  the  choices  people  make  and  the  degree  of  effort  they  will 
exert  (Keller,  1983).  Past  studies  indicate  that  motivation  is  affected  by  affective,  social,  and  cognitive  factors 
(Relan,  1992).  Keller  (1983;  1987a;  1987b)  identified  four  components  of  motivation  - i.e.,  attention,  relevance, 
confidence,  and  satisfaction  - and  strategies  to  design  motivating  instruction.  Clark  (1997;  1998)  developed  a 
CANE  (Commitment  And  Necessary  Effort)  model  that  identified  two  processes  of  motivation:  commitment 
and  necessary  effort.  Wlodkowski  (1993)  suggests  six  major  components  that  affect  adult  learners’  motivation 
in  the  time  continuum.  These  motivational  models  were  used  in  other  research  studies  to  identify  the  gap  in 
learner  motivation  and  how  to  design  motivating  instruction. 

Several  theories  have  provided  theoretical  frameworks  for  understanding  motivation  (Pintrich  & 
Schunk,  1996).  Among  different  constructs  on  motivation,  continuing  motivation  and  intrinsic  motivation  are 
the  most  significant  for  instructional  theory  and  research  (Kinzie,  1990).  Intrinsic  motivation  is  defined  as  the 
motivation  to  engage  in  an  activity  “for  its  inherent  satisfactions  rather  than  for  some  separable  consequence” 
(Ryan  & Deci,  2000).  Theories  of  motivation  and  empirical  evidence  have  suggested  several  sources  of  intrinsic 
motivation.  Some  motivational  researchers  posit  that  activities  that  provide  learners  with  a sense  of  control  over 
their  academic  outcomes  may  enhance  intrinsic  motivation  (Pintrich  & Schunk,  1996).  Lepper  and  Hodell 
(1989)  have  identified  challenge,  curiosity,  control,  and  fantasy  as  primary  characteristics  of  tasks  that  promote 
intrinsic  motivation. 

Continuing  motivation  is  the  type  of  intrinsic  motivation  most  directly  concerned  with  education  and  it 
reflects  an  individual’s  willingness  to  learn  (Maeher,  1976).  Studies  have  been  done  on  how  to  improve  learner 
motivation.  Theorists  argues  the  primary  reward  for  the  learner  is  the  activity  itself;  thus,  continuing  motivation 
is  facilitated  by  an  intrinsic  interest  in  the  activity  (Condry  & Chambers,  1978).  Similarly,  Merrill  (2002)  posits 
that  the  primary  reward  for  the  learner  is  learning  itself  - i.e.,  when  the  learner  is  able  to  show  a new  skill  or  an 
improvement  in  a skill,  he  is  motivated  to  perform  even  better.  He  suggests  it  as  an  integration  component  of 
effective  instruction. 

It  is  important  to  review  past  studies  on  motivational  issues  in  computer-assisted  instruction  and 
distance  education  settings,  since  motivational  features  encountered  these  settings  are  similar  to  those  in  Web- 
based  instruction  (Song,  2000).  Kinzie  (1990)  argues  that  intrinsic  and  continuing  motivation  are  important 
components  in  computer-based  instruction.  Malone  (1981)  suggests  challenge,  fantasy,  and  curiosity  as  the 
components  of  intrinsically  motivating  computer-based  instruction.  Song  (2000)  also  argues  that  three  types  of 
motivation  - motivation  to  initiate,  motivation  to  persist,  and  motivation  to  continue  - are  important  in  Web- 
based  instruction.  Studies  have  been  done  on  the  effects  of  delivery  medium  to  learner  motivation.  Several 
researches  suggest  that  motivation  to  learn  via  a particular  medium  is  influenced  by  the  learner’s  beliefs  about 
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his  own  ability  and  the  difficulty  level  of  the  task,  rather  than  by  the  medium  per  se  (Clark,  1994).  Similarly, 
Reinhart  (1999)  found  that  the  learner’s  self-efficacy  and  task  difficulty  affects  his  motivation  to  learn  via  the 
Web.  In  addition,  Keller  (1999)  posits  that  learner  support  is  important  for  motivating  learners  in  Web-based 
instruction. 


Methods 


Participants 

An  interview  research  method  was  used  as  a means  to  explore  the  issues  under  investigation. 
Interviews  were  conducted  of  adult  learners  who  have  taken  self-directed  online  courses.  The  sample  for  this 
study  was  drawn  from  over  3,000  working  adults  and  adult  students  who  registered  for  one  or  more  self- 
directed  online  courses  between  September  2003  and  January  2004,  which  was  retrieved  from  the  user  database 
of  the  company  who  were  the  provider  of  the  online  courses  that  the  study  participants  enroll.  From  this  user 
list,  about  200  people  were  purposefully  selected  to  get  a sample  representative  of  the  population  in  terms  of 
their  status,  gender,  experience  in  online  learning,  and  the  type  of  courses  taken.  The  adult  student  group  was 
drawn  from  students  enrolled  in  a large  Midwestern  university.  The  working  adult  group  was  drawn  from 
learners  in  three  different  types  of  organizations  (i.e.,  non-profit,  university,  and  business  organizations) 
throughout  the  U.S.  The  investigator  contacted  them  via  e-mail  soliciting  their  participation  in  the  study,  so  6 
working  adults  and  6 adult  students  agreed  to  participate.  Among  the  12  adults  interviewed,  7 were  females  and 
5 were  males.  The  participants  took  courses  of  various  topics;  seven  of  them  took  courses  on  desktop 
applications,  three  of  them  took  courses  on  computer  programming,  and  two  of  them  took  courses  on  soft  skills. 

Instrument 

A semi -structured  interview  method  was  used  to  collect  interview  data  in  this  study.  Several  open- 
ended  questions  were  asked  to  participants  to  explore  their  feelings  and  behaviors  with  regard  to  the  issues 
under  study.  The  interview  questions  consisted  of  three  parts;  introduction  to  the  interview,  leading  questions, 
and  the  concluding  session.  In  the  introduction,  the  purpose  of  the  study  and  information  on  the  confidentiality 
of  the  participant’s  responses  was  explained  to  him  or  her.  The  second  section  included  leading  questions  to  be 
asked  of  the  participant.  The  last  section  included  closing  remarks  on  thanks  the  participants  and  reminding 
them  of  possible  follow-up  questions  in  the  future. 

Two  pilot  interviews  were  administered  in  order  to  test  the  instrument  to  increase  the  clarity  and  the 
likelihood  of  eliciting  desired  information.  One  in-person  interview  and  one  phone  interview  were  conducted  in 
the  pilot  test  stage.  The  interview  questions  were  revised  based  on  the  feedback  from  this  pilot  study. 

Procedures 

One-on-one  interviews  were  conducted  from  March  through  June  of  2004.  8 participants  were 
interviewed  in  person  and  4 participants  who  were  located  at  distance  from  the  investigator  were  interviewed 
via  phone.  Semi  -structured  questions  were  asked  to  the  participants  to  explore  their  motivational  problems  and 
the  solutions  to  alleviate  the  problems.  In-person  interviews  were  held  either  at  a conference  room  or  the 
participant’s  office  room,  all  of  which  were  quite  rooms.  Each  interview  took  between  30  - 45  minutes  and  was 
tape  recorded.  The  interviews  were  transcribed  verbatim  for  analysis. 

A transcript  analysis  was  conducted  of  the  transcripts  in  order  to  identify  emerging  themes  or  patterns 
from  the  qualitative  data.  The  investigator  copied  the  transcripts  into  index  cards.  A sentence  or  sentences  that 
provided  information  relevant  to  the  research  questions  were  written  on  to  the  index  cards.  Two  investigators  - 
the  author  and  an  external  data  analyst  - sorted  the  index  cards  to  identify  emerging  themes.  The  index  cards 
were  sorted  by  grouping  the  cards  that  had  common  issues  or  topics  together.  Several  emerging  themes  were 
identified  as  a result.  An  external  auditor  also  reviewed  the  results  of  the  data  analysis  to  evaluate  the  provide 
feedback  on  the  trustworthiness  of  this  data  analysis. 

Results 

Motivation  to  start  online  instruction  (Motivation  to  initiate) 

Most  of  the  interview  participants  pointed  out  the  flexibility  and  convenience  of  self-paced  online 
learning,  (i.e.,  the  fact  that  they  can  learn  at  their  own  pace  without  the  time  constraints  of  classroom 
instructions  and  at  the  comfort  of  their  home)  as  the  primary  reason  for  choosing  an  online  training  option.  One 
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participant,  who  is  a full-time  working  professional  and  also  a part-time  graduate  student,  noted  that: 

I thought  it  was  a really  good  way  to  learn  more  software  programs  and  things  of  that  nature  that  I 
don’t  have  to  go  to  the  classroom  one  as  a part-time  student,  full-time  mother,  full-time  worker.  It’s 
just  easier,  and  I can  do  it  from  home.  You  know,  I don’t  have  to  be  on  campus,  it  just  takes  my  user 
name  and  password,  and  I can  do  this  from  home  without...  any  time  I want  to.  I mean,  you  know,  I’m 
not  restricted  to  time  or  anything,  and  I really  like  that.  And  if  I don’t  have  time  to  finish  something,  it 
will  save  my  spot,  and  I like  that  a lot,  considering  my  interruptions. 

Motivational  changes  and  learner  persistence 

Although  the  convenience  and  flexibility  of  self-paced  online  learning  was  the  biggest  motivator  for 
them  to  choose  self-paced  online  training  options,  it  did  not  necessarily  motivate  them  enough  to  persist  in  their 
learning.  Participants’  retention  rates  were  investigated  as  an  indicator  of  learner  persistence  (i.e.,  the 
motivation  to  persist).  The  results  indicated  that  half  of  the  participants  did  not  complete  the  courses,  as  seen  in 
Table  1.  There  were  also  a different  range  of  retention  rates  across  subject  areas  - i.e.,  0%  of  those  who  took 
computer  programming  courses  completed  the  course,  whereas  100%  of  those  who  took  soft  skill  courses 
completed  the  course  (see  Table  1). 


Table  1 .Retention  rates  of  the  participants  of  self-paced  online  courses 


Topics 

# of  Participants 

# of  Drop-outs 

Retention  rates 

Computer  programming 

3 

3 

0% 

Desktop  applications 

7 

3 

57% 

Soft  skills 

2 

0 

100% 

Total 

12 

6 

When  asked  why  they  did  not  complete  the  course,  three  adult  students  indicated  they  did  not  complete 
the  course  because  it  was  too  boring.  Two  working  adults  indicated  that  lack  of  time  was  the  main  reason  that 
they  did  not  complete  the  course.  One  adult  student  even  failed  to  start  the  instruction  due  to  the  difficulty  of 
navigation. 

When  asked  about  their  motivational  change,  eight  out  of  the  twelve  people  interviewed  indicated  that 
their  motivational  level  did  not  change.  Four  of  them  stated  that  their  motivation  waned  as  they  went  through 
the  instruction.  The  lack  of  interaction  (both  computer  interaction  and  human  interaction)  in  their  learning  was 
the  major  reason  for  them  to  get  bored  with  the  instruction,  and  ultimately  waned  their  persistence.  One 
participant  described  her  motivational  change  as  follows: 

At  first,  I think  that  I’m  really,  really  excited  and  I want  to  do  this,  and  I get  all  into  it,  and  then  after 
when,  I would  say,  half-way  through,  I get  sort  of  not  as  motivated  - maybe  even  a little  bit  bored  with 
it  to  a certain  extent...  The  convenience  is  nice,  but  that’s  not  what  keeps  it.  It  makes  you  want  to  try 
it,  but  it’s  not  what  keeps  you  interested  in  it.  It’s  got  to  have  more  interaction.  It  doesn’t  hold  my 
interest  as  long  as  what  I think  it  should,  and  I think  if  there  was  some  more  interactivity  of  a program, 
then  it  would  really  keep  my  interest  more,  and  I would  be  more  enthused  about  taking  more  courses. 

Interactivity  (human-computer  interaction)  during  online  instruction 

When  asked  what  motivated  or  demotivated  them  to  learn  while  they  took  the  self-paced  online  course, 
participants  stated  that  animations  and  simulations  in  the  online  courses  were  interactive  and  that  those 
interactive  features  helped  them  engaged  in  their  learning.  One  participant  who  took  an  introductory  course  on  a 
computer  application  (Macromedia  Dreamweaver)  stated  that: 

The  one  thing  I did  like  about  Dreamweaver  (course)  is  it  was  very  interactive  and  it  would  let  you 
move  things  within  while  it’s  doing  it,  but  the  other  one  that  I had  tried  to  take  before  wasn’t  like  there. 
Oracle  (course)  was  the  one  I took  before,  that,  it  was  just  you  read  and  then  you  try  and  answer  the 
questions  about  what  you  read.  If  it  was  just  read  to  answer  questions,  I mean,  my  interest  in  that 
subject  went  down  within  an  hour.  I didn’t  want  to  do  it  anymore.  If  it  was  interactive  like  the  one 
with  Dreamweaver  (course)  where  it  said  to  click  the  button  and,  you  know,  you  could  see  what  it  did, 
then  I was  more  interested  in  that.  But  if  it  was  read  to  answer  questions.  I’d  get  really  sleepy  and 
bored. 

As  was  illustrated  in  the  participant’s  comment  above,  the  lack  of  interactivity  made  them  lose  interest 
in  the  topic  and  was  a major  reason  for  dropping  out  of  the  course,  in  particular,  among  those  who  took 
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computer  programming  courses. 


Human  interaction  during  online  instruction 

The  effects  of  human  interaction  on  the  learners’  motivation  appear  to  be  different  according  to  the 
context  they  are  in.  Most  of  the  adult  students  (4  out  of  6)  participated  in  this  study  mentioned  that  human 
interaction  is  important  for  their  learning  and  the  lack  of  human  interaction  in  this  kind  of  online  learning 
environment  could  probably  decreased  their  motivation  to  persist  in  their  learning.  For  example,  a participant 
who  is  a part-time  graduate  student  mentioned  that: 

1 don’t  think  that  you  have  to  have  a classroom  to  have  learning  take  place.  You  have  to  have 
interaction,  though,  whether  it’s  through  e-mails,  whether  that  interaction  takes  place  through  phone 
calls,  getting  together  at  the  coffee  shop,  however  that  interaction  takes  place,  because  it  does,  at  least 
for  my  personality,  it  does,  1 think  that  interaction  does  help  the  learning  process.  It  helps  to  motivate 
you.  You’re  not  like,  you  know,  separate,  you’re  out  here  in  this  little  world  and  the  entire  world’s 
over  here,  but  being  part  of  that  group,  it  does  actually  help  you  in  the  learning  process.  You  can  see 
where  you’re  going. 

In  contrast,  working  adults  responded  that  human  interaction  was  not  important  for  their  learning  in 
this  kind  of  learning  environment  and  therefore  the  lack  of  human  interaction  had  little  impact  on  their 
motivation.  To  them,  flexibility  in  their  learning  was  more  important  than  having  an  instructor  for  their 
learning,  as  noted  by  a participant  who  was  a full-time  working  adult  as  follow: 

It  would  depend  on  the  time  flexibility,  the  ability  to  do  it  at  any  time  was  probably  more  important  in 
this  particular  case  than  having  an  instructor.  If  I could  do  the  same  thing  as  a structured  class  and  had 
the  time  I would  probably  prefer  having  an  instructor  but  this  being  a fairly  small  class,  a fairly  small 
unit  of  material  and  having  the  flexibility  whenever  I wanted  to  was  definitely  a plus  rather  than 
having  to  schedule  a particular  time  to  be  at  a particular  place  or  be  online  at  a particular  time  or 
whatever. 

One  interpretation  for  such  differences  between  adult  students  and  working  adults  on  their  perceptions 
of  the  lack  of  human  interaction  is  that  working  adults  seemed  to  be  more  independent  learners  than  those  in  the 
school  setting.  Three  out  of  six  working  adults  interviewed  mentioned  that  they  would  prefer  to  learn  in  a self- 
paced  format  over  an  instructor-led  one.  For  instance,  one  working  adult  stated  the  reason  that  why  she  chose  an 
online  training  option  over  classroom  instruction  as  follows: 

It  (online  training)  was  probably  less  boring.  I could  control,  part  of  it  is  I can  control  my  learning 
experience  better.  Take  it  in  bits  and  pieces  and  not  have  to  spend  all  day  in  one  class  with  a teacher 
who  might  not  be  very  good.  I would  just  rather  learn  it  myself. 

In  contrast,  four  out  of  six  adult  students  interviewed  mentioned  that  presence  of  an  instructor  would 
help  their  learning  process  (i.e.,  being  able  to  ask  questions).  For  example,  a graduate  student  who  took  a course 
on  statistical  computer  program  mentioned  that: 

For  instance,  with  in  say  a two-way  ANOVA  design  or  something  like  that  and  you’re  partitioning 
sums  of  squares  a certain  way  you  know  I understand  how  to  do  it  but  I don’t  necessarily  understand 
why  I have  to  do  it  and  a person  could  help  explain  to  me  why  I had  to  do  so  perhaps  structuring  it  that 
way  where  there  is  there  is  the,  where  there’s  the  tools  component  and  guiding  me  through  how  to  do  it 
and  then  a person  kind  of  suggesting  well  this  is  why  we’re  doing  what  we’re  doing.  For  me  that 
would  be  really  helpful  and  then  since  that  wasn’t  there  I was  kind  of  left  to  my  own  devices  to  kind  of 
try  to  understand  why.  Now  verbally  you  could  say  why  but  perhaps  for  me  it  helps  if  somebody  is 
telling  me  that. 

Application  and  integration  of  content  by  the  learner 

In  response  to  questions  that  what  engaged  or  interested  them  in  their  online  learning,  the  participants 
indicated  that  they  were  interested  in  activities  that  simulate  real-world  situations  and  give  them  hands-on 
experience,  such  as  animations  and  simulations.  These  instructional  approaches  are  known  to  be  effective,  and 
they  also  seem  to  motivate  learners  as  well.  One  participant  who  took  a course  on  computer  application 
described  his  experience  engaging  in  animations  in  the  course  as  follows: 

The  thing  I liked  about  the  course  was  the  fact  that  they  had  actual  parts  of  the  program  in  it.  And  I 
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think,  it  looks  like  they  set  it  up,  obviously  it  wasn’t  actually  the  program,  but  they  just  had  areas 
where  if  you  were  as  close  to  you  if  you  were  actually  doing  the  work  in  the  program  and  I actually 
opened  up  the  program  and  did  stuff  that  way,  too.  So  I could  actually  find  out  how  it  worked.  I 
thought  it  was  good.  I thought  it  was  good  that  they  went  through  and  used  the  actual  program. 


In  addition  to  animations,  simulations  also  seemed  to  be  engaging  and  interesting  for  the  learners  as  mentioned 
by  a participant,  who  was  a full-time  working  professional  and  took  a course  on  consulting  skills: 

The  simulations  were  also  very  good.  The,  they’re  very  rich  in  where  you  like  if  you’re  having  a,  you 
can  have  a conversational  simulation  with  someone  and  actually  sort  of  an  intelligent  conversation 
where  you  are  asked  to  respond  appropriately  and  then  it  scores  you  on  how  well  you,  which  response 
was  more  appropriate.  And  I really,  it’s  kind  of  like  game  playing.  It  was  fun  for  me  to  try  to  guess 
what  was  a more  appropriate  response. 

Learner  control 

Most  of  the  participants  indicated  that  they  felt  positive  about  being  able  to  control  the  pace  of  their 
learning.  One  participant  who  was  a part-time  student  with  a full-time  job  and  took  a course  on  computer 
applications  noted  that: 

...and  the  fact  that  it’s  work  at  your  own  pace  is  nice,  too.  Because  there  are  some  online  classes 
where  they  do  want  you  to  be  on  a schedule,  to  have  things  turned  in  if  you’re  being  graded  and  so 
forth,  but  with  this  online  course,  it’s  nice  to  just  find  the  time,  find  a half  hour  here  or  there  and  go  in 
and  work  on  it,  and  not  feel  that  pressure  of  I have  to  do  this  right  away. 

Most  of  the  participants  also  preferred  the  control  over  the  sequence  of  instruction  so  that  they  could 
skip  the  part  that  they  are  already  familiar  with  and  spend  more  time  on  the  part  they  are  not  familiar  with.  One 
participant  stated  that: 

The  one  thing  I guess  I kind  of  preferred  a difference  between  the  two  (face-to-face  and  online 
instruction)  was  that  when  we  were  covering  sections  that  I was  familiar  with  I was  able  to  move 
through  very  quickly  versus  a classroom  I would  have  had  to  sit  through  their  lesson  outline  for  them 
to  get  through  that.  I was  able  to  skip  over  the  easier  things  and  go  to  the  harder  parts  I really  wanted 
to  spend  time  on. 


Conclusion  / Discussions 


Implications  of  the  Findings 

The  results  of  this  study  confirm  other  research  findings  that  the  lack  of  motivation  is  the  major  reason 
for  student  drop-outs  in  online  courses.  It  also  provides  empirical  support  for  the  claims  by  theorist  that  three 
types  of  motivation  (i.e.,  motivation  to  start,  motivation  to  persist,  and  motivation  to  continue)  can  influence 
learners’  motivation  in  self -directed  online  learning  settings,  those  motivation  can  change  over  time  as  learners 
go  through  instruction.  Given  the  findings  of  this  study  that  the  learner’s  motivational  level  changes  as  they  go 
through  instruction,  it  is  suggested  that  instructional  designers  need  to  put  various  factors  into  account  that 
influences  the  learner’s  persistence  and  continuing  motivation. 

This  study  found  several  factors  that  Interaction  is  found  to  be  critical  for  creating  motivating  online 
learning  environments.  The  learners  felt  that  computer-learner  interaction  is  critical  for  motivating  online 
learning  environments.  Yet,  the  lack  of  human  interaction  in  the  self-paced  online  learning  environment  did  not 
seem  to  impact  the  motivation  of  adults  in  workplace  learning  settings. 

The  findings  of  this  study  also  provide  some  implications  for  the  design  of  motivating  self-paced 
online  learning  environments.  This  study  found  that  aninimations  and  simulations  are  beneficial  in  engaging 
learners  in  self-paced  online  learning  environments.  Also,  the  application  of  content  to  real  world  situations 
were  found  to  be  motivating  to  learners.  Such  an  approach  is  regarded  as  effective  instruction  by  many 
researchers  and  it  is  also  seem  to  be  an  effective  way  to  motivate  learners. 

Limitations  of  the  Study 

It  should  be  acknowledged  that  there  are  some  limitations  to  this  study.  This  study  adopted  a case 
study  approach  in  the  sense  that  it  investigated  learners  of  online  courses  developed  by  a particular  e -learning 
vendor.  Since  there  are  online  courses  developed  by  many  other  e-learning  vendors  and  since  their  courses  are 
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not  designed  in  the  same  way,  it  is  recommended  that  the  self-paced  online  courses  developed  by  several  e- 
learning  vendors  are  investigated  to  enhance  the  generalizability  of  the  findings  of  this  study. 

Since  this  study  examined  on  self-directed  online  course  format,  it  is  likely  that  the  findings  of  this 
study  might  be  limited  to  this  particular  type  of  online  learning  environments.  Therefore,  readers  should  caution 
not  to  generalize  the  findings  to  other  types  of  online  courses  (e.g.,  formal  distance  education  programs).  Also, 
since  this  study  was  conducted  of  adult  learners  (e.g.,  adult  students  and  working  adults),  their  motivational 
needs  might  be  different  from  those  of  school  children  or  young  adults,  as  suggested  by  several  motivational 
theorists.  Therefore,  it  is  suggested  that  the  findings  of  this  study  might  not  be  generalized  to  younger  age 
groups.. 

Recommendation  for  Future  Studies 

This  study  was  a qualitative  case  study  in  which  the  aim  was  to  explore  the  issues  under  study  in 
details  with  a relatively  small-sample  of  subjects.  Therefore,  it  is  recommended  that  a quantitative  study  be 
conducted  with  a laiger  sample  to  enhance  the  generalizability  of  the  findings  of  the  study.  Also,  it  is  my 
speculation  that  different  factors  might  influence  learner  motivation  in  different  types  of  online  learning 
environments  and  with  different  kinds  of  learner  characteris  tics.  Since  this  study  is  conducted  of  learners  who 
took  a particular  type  of  online  courses  (i.e.,  self -directed  online  courses),  it  is  recommended  that  the  studies  are 
conducted  on  what  influences  learner  motivation  in  different  online  learning  environments.  Such  studies  will 
provide  insights  on  whether  the  findings  of  the  study  can  be  applied  to  other  types  of  online  learning 
environments  (e.g.,  instructor-led  online  courses,  online  degree  programs). 
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Introduction 

Online  teaching  and  learning  is  making  a significant  impact  on  the  fabric  of  higher  education.  In 
particular,  online  MBA  programs  have  seen  a rapid  rise  in  student  enrollments  in  recent  years  while  the  student 
enrollments  in  traditional  in -residence  MBA  programs  are  in  decline  (Hayward,  2004;  Lorenzo,  2004).  This 
appears  to  be  due,  in  part,  to  the  convenience  and  flexibility  of  the  delivery  of  online  education,  which  enable 
adults  with  full-time  jobs  to  attend  classes  without  having  to  leave  their  current  jobs  (Mangan,  2001). 
Accordingly,  many  institutions  of  higher  education  are  offering  online  programs  to  serve  the  growing  learner 
population.  Despite  such  an  increase  in  the  popularity  of  online  education,  there  has  been  a concern  in  the 
quality  of  online  education  (Diaz,  2002;  Islam,  2002;  Moore  & Kearsley,  1996).  Therefore,  institutions  of 
higher  education  have  keen  interests  in  offering  quality  online  MBA  programs  . Moreover,  educators  need  to  be 
aware  that  student  expectations  on  the  quality  of  online  education  programs  are  rising  rapidly  (Bonk,  2004). 

As  online  learning  has  gained  its  wide  acceptance  in  higher  education,  there  is  an  increasing  awareness 
of  the  facilitative  role  of  online  instructors.  While  there  is  growing  interest  in  examining  instructor  online 
facilitation  roles  in  distance  learning,  few  empirical  studies  have  examined  the  issues  and  challenges  the 
instructor  are  faced  with  when  facilitating  online  courses.  This  gap  in  the  research  is  especially  apparent  in 
facilitating  online  learning  communities  and  teamwork.  The  results  reported  in  this  study  extend  previous 
findings  regarding  the  moderating  roles  of  online  instructors  (Anderson,  Rourke,  Garrison,  & Archer,  2001; 
Berge,  1995;  Feenberg,  & Xin,  2002)  and  learning  communities  (Eastmond,  1995;  Lave  & Wenger,  1995; 

Oram,  1998).  In  addition,  they  represent  initial  findings  obtained  from  a long  term  research  project  intended  to 
enhance  the  quality  of  instruction  in  a fast-growing  online  MBA  program. 

This  paper  presents  preliminary  results  of  a study  of  an  accredited  online  MBA  program  at  a top 
ranked  business  school  in  a large  Midwestern  university.  323  students  enrolled  in  the  public  online  MBA 
program  during  the  academic  year  of  2004-2005.  This  particular  business  school  under  investigation  also  offers 
corporate  online  MBA  programs,  which  are  offered  in  partnership  with  business  organizations,  and  403  students 
were  enrolled  in  corporate  online  MBA  programs  in  2004.  This  online  MBA  program  has  grown  to  include 
hundreds  of  students  in  just  a few  years.  Although  this  program  has  maintained  a very  high  student  retention 
rate  over  the  past  years  since  its  inception,  there  was  a need  for  a systematic  approach  to  evaluating  the  program 
due  to  the  low  response  rate  of  course  evaluations  from  the  students.  The  purpose  of  this  study  is  to  explore  the 
impact  of  online  facilitation  as  an  effective  instructional  tool  on  several  key  components  of  online  teaching  and 
learning:  online  learning  activities,  social  presence  and  learning  community,  and  virtual  teamwork.  To  this 
intent,  this  study  will  focus  on  the  following  research  questions: 

• What  are  the  key  strategies  that  instructors  have  used  in  facilitating  online  learning  and  motivating 
student  learning? 

• How  do  students  perceive  the  effectiveness  of  instructors’  online  facilitations? 

• What  are  the  challenges  and  issues  confronting  instructors  in  facilitating  an  online  learning 
environment? 

The  results  of  this  study  are  expected  to  provide  implications  for  the  improvement  of  the  online  MBA 
program  not  just  for  the  one  under  investigation  but  for  other  online  MBA  programs  as  well.  In  particular,  it  is 
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hoped  that  this  study  will  help  provide  a better  understanding  of  the  issues  related  to  teaching  and  learning  in 
online  MBA  courses. 


Literature  Review 

Despite  the  increased  importance  on  the  role  of  the  learner  in  learner-centered  approaches  in  online 
learning  environments,  many  researchers  suggest  that  the  instructor  still  plays  important  roles  in  online  learning 
(Arbaugh,  2000).  Several  studies  have  conducted  of  how  online  learning  should  be  designed  and  facilitated. 
Many  researchers  posit  that  online  discussions  in  asynchronous  learning  environments  foster  in-depth  and 
critical  thinking  of  students  by  allowing  them  the  time  to  process  their  thinking  when  they  post  a message  in 
online  conferences  (Duffy,  Dueber,  & Hawley,  1998).  Bonk,  Hansen,  Grabner-Hagen,  Lazar,  and  Mirabelli 
(1998)  suggest  that  asynchronous  conferencing  was  the  preferable  method  for  fostering  in-depth  student  online 
discussions  and  rich  interactions  than  synchronous  conferencing  among  preservice  teachers.  Benbunan-Fich  and 
Hiltz  (1999)  found  in  their  study  of  case  studies  through  asynchronous  learning  networks  in  an  online  MBA 
course  that  students  participated  in  an  asynchronous  learning  environment  were  able  to  produce  better  and 
longer  solutions  to  the  cases  than  the  students  participated  in  in-class  discussions,  but  the  online  students  were 
less  satisfied  with  the  interaction  process.  Several  other  studies  also  report  positive  results  of  using 
asynchronous  online  discussions  to  facilitate  case  studies  in  online  MBA  classes  (Benbunan-Fich  & Hiltz,  1999; 
Rourke  & Anderson,  2002;  Henson,  Kennett,  & Kennedy,  2003). 

Nevertheless,  past  studies  also  suggest  that  there  are  challenges  in  facilitating  student  learning  through 
online  interactions.  Computer- mediated  communications,  especially  in  text -based  conferencing,  seem  to  a 
limited  mode  of  communication  compared  to  face-to-face  communications  in  terms  of  the  richness  of 
communication  modes  that  can  be  used  (Curtis  & Lawson,  2001;  Draft  & Lengel,  1984).  Also,  Herring  (1999) 
found  that  there  is  a high  degree  of  disrupted  adjacency,  overlapping  exchanges,  and  topic  decay  in  computer- 
mediated  communications  both  in  asynchronous  and  synchronous  conferencing  settings. 

The  design  of  online  courses  is  important  for  the  success  of  online  learners.  Various  methods  of 
instruction  have  been  applied  to  teaching  business  courses  online.  Case-based  learning  has  been  a dominant 
method  of  teaching  in  many  of  the  courses.  In  particular,  authenticity  and  relevance  of  the  cases  seemed  to  be 
critical  for  students’  engagement  in  their  learning  through  cases  (Henson,  Kennett,  & Kennedy,  2004;  Theroux, 
Carpenter,  & Kilbane,  2004).  Online  MBA  course  can  foster  students’  reflective  thinking  (Ascribe,  2004:  Hay, 
Peltier,  & Drago,  2004).  Also,  creating  virtual  communities  among  online  MBA  students  and  fostering 
interactions  among  them  made  a significant  impact  on  the  students’  evaluation  of  the  effectiveness  of  their 
online  learning  experience  (Hay,  Hodgkinson,  Peltier,  & Drago,  2004;  Paltier,  Drago,  & Schibrowsky,  2004). 

Creating  virtual  teams  are  also  of  concern  in  facilitating  online  courses.  There  is  an  increasing  interest 
in  the  learning  theory  that  stresses  learning  as  a social  activity  (Barab  & Duffy,  2000;  Jonassen,  2002).  Some 
researchers  argue  that  knowledge  is  situated  in  social  practices;  therefore,  knowledge  can  be  acquired  in  the 
context  that  it  is  actually  practiced.  From  this  viewpoint,  students  can  acquire  the  competence  of  experts  by 
participating  in  the  practice  of  the  community  by  engaging  themselves  into  the  activity  and  culture  of  the  group 
that  the  knowledge  and  skills  are  practiced  (Barab  & Duffy,  2000;  Brown,  Collins,  & Duguid,  1989;  Lave  & 
Wenger,  1991). 

Studies  have  been  conducted  on  characteristics,  needs,  and  concerns  of  online  learners.  Many  studies 
suggest  that  learners  can  learn  in  online  settings  as  effectively  as  in  face-to-face  settings.  Many  researchers 
argue  that  online  learning  is  not  for  everyone  and  suggest  that  different  learning  styles  need  to  be  addressed. 
Self- motivated  learners  are  more  likely  to  succeed  (McCall,  2002).  Students’  experience  with  online  learning 
appears  to  be  an  important  factor  in  their  perceptions  of  learning  and  satisfaction.  In  a study  of  online  MAB 
students,  Arbaugh  and  Duray  (2002)  found  that  students  who  had  more  experience  in  online  learning  were  more 
likely  to  be  satisfied  with  learning  over  the  Internet.  Conrad  (2002)  also  found  from  her  study  of  undergraduates 
students  that  students  who  had  more  experience  in  online  courses  were  less  likely  to  feel  anxious  about  online 
learning. 


Methodology 

A case  study  approach  was  used  for  this  study.  A case  is  “instrumental”  (Stake,  1994)  in  providing  an 
understanding  of  the  issues  of  how  to  facilitate  learning  online.  Data  collected  in  this  study  included:  (1)  semi  - 
structured  one-on-one  interviews  with  selected  faculty  members  and  students;  (2)  surveys  of  the  instructors  and 
students  on  their  perceptions  of  the  issues  in  online  teaching  and  learning;  (3)  in  -person  focus  group  interviews 
with  the  instructors  and  the  students;  and  (4)  content  analyses  of  course  documents  and  class  assignments 
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including  student  participation  in  class  activities  posted  in  the  course  management  system. 

Interviews  were  conducted  of  the  instructors  and  the  students  of  the  online  MBA  program  for  an  in- 
depth  understanding  of  the  issues  under  study.  Subsequent  surveys  were  conducted  in  order  to  enhance  the 
generalizability  of  the  findings  of  the  study.  In  addition,  a content  analysis  was  conducted  of  the  online  courses 
in  order  to  explore  how  the  courses  were  designed  and  also  to  triangulate  the  data  with  those  from  the 
interviews  and  the  surveys.  More  detailed  descriptions  of  the  methods  of  this  study  are  presented  below. 

The  Interview  Study 

26  faculty  members  of  the  online  MBA  program  and  3 faculty  members  who  taught  traditional 
residence  MBA  courses  only  were  interviewed  in  person.  25  semi -structured  questions  were  asked  and  the 
interview  was  conducted  in  an  one-on-one  interview  format.  The  interview  questions  included  the  participants’ 
background  information,  their  perceptions  of  the  issues  on  delivering  case-based  learning  online,  interaction  in 
online  settings,  learner  control,  collaboration  and  motivation,  and  the  online  learning  environment  in  general. 
Each  interview  took  45  - 60  minutes. 

In  addition  to  the  faculty  interviews,  40  online  MBA  students  were  also  interviewed.  20  of  them  were 
first  year  students  who  were  participating  in  a one-week  orientation  program  held  on  campus.  The  other  20  of 
them  were  second  year  students  who  were  also  participating  in  a one-week  orientation  program  held  on  campus. 
Each  interview  took  place  in-person  in  a meeting  room  on  the  campus  and  took  3045  minutes.  In  addition  to 
the  one-to-one  student  interviews,  20  students  participated  in  four  focus  group  interview  sessions  (e.g.,  five 
students  per  sessions)  held  during  their  orientation  program.  The  focus  interviews  were  also  held  in  a meeting 
room  on  the  campus  and  took  45  - 60  minutes.  A different  set  of  questions  were  asked  to  first-year  and  second- 
year  students  because  first-year  student  were  interviewed  right  before  their  first  semester  began  thus  had  no 
experience  with  online  MBA  courses  at  the  time  of  the  interview.  In  contrast,  the  second-year  students  had 
finished  their  first-year  of  the  program  and  was  about  to  start  their  second  academic  year  in  the  program. 
Therefore,  the  interview  questions  for  the  first-year  students  focus  on  their  expectations  on  the  online  MBA 
program,  whereas  the  interview  questions  for  the  second-year  students  focused  on  their  experience  taking  online 
MBA  courses.  Every  interview  session  was  taped  recorded  and  later  transcribed  for  analysis. 

For  qualitative  data  analysis,  a constant  comparative  method  was  used  to  triangulate  the  data  from 
different  data  sources  and  to  identify  emerging  themes.  Multiple  researchers  were  involved  to  test  the  coding 
reliability.  Member  checking  was  also  used  to  ensure  the  trustworthiness  of  the  study. 

The  Survey  Study 

First-year  and  second-year  public  online  MBA  students  were  surveyed.  The  survey  instrument  for  first- 
year  online  MBA  students  consisted  of  47  questions  regarding  their  expectations  on  taking  online  MBA 
courses.  This  survey  instrument  included  multiple  choice  questions,  5-point  scale  Likert  type  questions,  and 
some  open-ended  questions.  The  survey  instrument  for  second-year  online  MBA  students  consisted  of  67 
questions  regarding  their  perceptions  and  attitudes  toward  the  online  MBA  program. 

The  paper-based  questionnaires  were  handed  out  to  the  students  who  were  present  at  the  one-week 
program  orientation  session  held  on  campus  in  the  summer  of  2004.  162  first-year  students  and  102  second-year 
students  returned  the  survey,  which  accounted  for  an  almost  100%  return  rate.  The  students  participated  in  this 
study  had  various  backgrounds  in  terms  of  their  age,  gender,  location,  professional  experiences,  and  online 
learning  experiences.  Among  those  students  surveyed,  40  of  them  also  participated  in  one-one-one  or  focus 
group  interviews  prior  to  or  after  the  survey. 

The  instrument  for  faculty  survey  consisted  of  65  questions  and  was  divided  into  three  sections.  The 
first  section  of  this  survey  instrument  asked  the  participants’  background  information.  The  second  section  of  the 
questionnaire  consisted  of  5-point  scale  Likert  type  questions  about  their  overall  perceptions  and  attitudes 
toward  teaching  online  MBA  courses.  The  last  section  included  four  open-ended  questions  soliciting  the 
participants’  general  comments  about  teaching  online  MBA  courses  and  how  to  improve  the  online  MBA 
program.  The  faculty  survey  instrument  was  developed  on  the  Web  using  a Web-based  survey  tool. 

For  data  analysis,  the  survey  data  was  entered  into  SPSS  for  analysis.  Various  statistical  analyses, 
including  descriptive  statistics  and  correlational  analyses  were  employed  for  the  data  analysis. 

Content  Analysis 

27  online  MBA  courses  across  various  business  disciplines  were  selected  for  content  analysis.  The 
content  of  course  Web  sites  were  analyzed  based  on  a coding  scheme.  The  coding  scheme  included  following  9 
categories:  (1)  course  structure,  (2)  interactivity,  (3)  social  interaction,  (4)  instructional  design  support,  (5) 
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instructor  facilitation,  (6)  case  type,  (7)  learner  collaboration,  (8)  degree  of  learner  control,  (9)  assessment.  In 
addition,  10  subjective  rating  scales  were  developed  to  assess  the  level  of  interactivity  in  the  online  courses 
under  investigation. 

Descriptive  data  were  obtained  by  counting  the  frequencies  of  occurrences  based  on  the  coding 
scheme.  Two  investigators  analyzed  the  data  independently  and  compared  with  each  other  later  to  check  for 
inter-reliability  of  the  data  analysis. 


Results 

This  study  is  still  in  progress  at  this  point  and  analyses  of  student  and  faculty  interviews  data  and 
student  survey  data  are  completed  so  far.  The  results  of  faculty  interviews  and  student  surveys  are  presented 
below. 

Content  Analysis 

82%  of  the  courses  analyzed  used  asynchronous  class  discussions  (e.g.,  discussion  forums),  whereas 
44%  of  the  courses  had  synchronous  class  discussions  (e.g.,  chat  rooms).  33  of  those  courses  had  both 
asynchronous  and  synchronous  class  discussions.  A majority  of  the  courses  under  investigation  (81%)  used 
team  activities  in  their  courses,  whereas  19%  of  those  courses  did  not  use  any  team  activities  and  had  only 
individual  activities.  Students  chose  their  team  in  48%  of  the  courses  under  investigation,  and  instructors 
assigned  teams  in  33%  of  those  courses.  20  out  of  the  27  courses  being  studied  used  cases  to  support  student 
learning. 

Faculty  Interviews 

Online  Facilitation  through  Asynchronous  Interactions 

The  study  found  that  the  online  instructors  use  asynchronous  tools  more  often  than  synchronous  tools 
to  facilitate  students  online.  The  instructors  mentioned  that  they  used  e-mail,  announcements  in  the  course 
management  system,  and  asynchronous  discussion  forums  as  primary  ways  to  facilitate  students.  They  used 
discussion  questions  (open  up  a discussion  by  asking  a question,  prompting  questions  for  further  questions), 
recognizing  good  points,  and  summarizing  discussions  as  the  techniques  that  they  use  to  facilitate  online 
discussions. 

Most  of  the  instructors  valued  the  immediacy  of  instructor’s  feedback  to  students  to  respond  to  the 
needs  of  the  online  learners  ’ need  promptly  Also,  the  instructors  noted  that  it  was  critical  for  them  to  make  sure 
the  equal  participation  of  the  students  in  online  courses  for  effective  facilitate  of  online  courses.  The  following 
is  a quote  from  an  instructor  of  an  operation  management  course  that  illustrates  the  importance  of  providing 
guidelines  to  ensure  students’  equal  opportunities  in  contributing  to  the  discussion: 

So  this  year,  I said  [to  the  students]  you  can  only  contribute  on  the  first  day  to  two  of  the  cases; 
you  can’t  contribute  to  all  four.  Because  there  were  guys  in  the  previous  years  that  would  try  to 
beat  everybody  to  the  punch  on  the  cases,  and  answer  every  question  on  every  case,  and  it  got 
some  of  the  others  upset,  so  I said,  no,  you  can  only  post  to  two.  And,  I give  them  some 
suggestions  on  these  things.  You  don’t  have  to  answer  every  question.  I’m  interested  in  quality, 
not  quantity.  This  time,  they  were  much  more  disciplined. 

Online  Facilitation  through  Synchronous  Interactions 

The  instructors  mentioned  some  barriers  in  using  synchronous  tools  to  facilitate  their  online 
course.  Almost  every  instructor  tried  to  use  chat  rooms  in  facilitating  online  discussion  or  holding  office 
hours  in  the  beginning.  However,  a majority  of  the  instructors  discontinued  using  synchronous  tools  for 
several  reasons,  which  were  found  from  the  analysis  of  interview  transcripts.  First,  the  limited  functions 
of  chat  room  tools  in  the  course  management  system  presented  a major  barrier  (e.g.,  small  text  input  box 
broke  a large  chunk  of  text  input  by  the  instructors  or  the  students).  Secondly,  it  was  extremely  difficult 
to  schedule  a time  for  all  group  participants  to  attend  a synchronous  chat  session  when  the  participants 
were  located  in  different  regions  of  the  United  States  or  the  world.  Consequently,  the  participation  rate  of 
real-time  chats  was  often  fairly  low  and  both  the  students  and  instructors  were  not  satisfied  with  the 
learning  experience.  Finally,  the  instructors  realized  their  lack  of  moderating  skills  and  experiences  when 
the  size  of  the  conference  reached  above  ten  people  as  well  as  the  constraints  of  their  typing  speed,  as  an 
instructor. 

However,  a few  faculty  members  who  used  chat  room  found  it  helpful.  Two  instructors  mentioned  the 
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convenience  of  bringing  guest  experts  to  the  chat  room,  as  noted  in  the  following  comment  by  an  instructor: 
What  I try  to  do  every  week  is  have  a chat  session  and  I find  those  usually  work  pretty  well  and 
last  semester  when  I taught  this  course,  actually  it’s  a different  course  in  the  winter  quarter,  I did 
something  I hadn’t  done  before  which  I brought  in  some  outside  speakers  for  the  chat  session. 

So  since  these  guys  can  do  this  from  anywhere  I had  the  executive  vice  president  of  Intel  online 
one  week  and  I had  the  CFO  of  this  company  called  Finish  Line.  So  I think  the  students  really 
liked  that.  I had  never  really  thought  about  doing  that  before  but  it’s  very  easy. 

Needs  for  Better  Technology  and  Tools  for  Online  Facilitation 

Approximately  half  of  the  instructors  interviewed  mentioned  that  that  they  would  be  interested  in 
trying  some  advanced  technology  tools . Some  instructors  noted  that  more  real-time,  visual-based  learning  tools 
with  multiple  modes  of  communication  channels  to  enrich  online  learning  environment  and  further  improve  the 
efficiency  and  effectiveness  of  online  learning.  With  those  tools,  it  was  expected  that  it  could  help  establish  a 
better  professional  intimacy,  realism,  and  real-world  flavor  for  online  learning  participants.  Such  visual 
representation  and  hands-on  tools  were  perceived  lacking  in  existing  online  courses  in  this  program,  as  noted  in 
the  following  comments  by  two  instructors: 

For  example,  if  I’m  doing  a lecture  on  inventory  costs,  I think  it  would  be  useful  using 
whiteboard  technology  and  a video  camera  to  have  me  do  what  I would  do  in  a classroom, 
maybe  not  for  everything  but  for  key  points,  actually  go  through  things  on  the  board,  or  go 
through  a problem  where  the  problem  itself  appears  in  a window  on  the  video  screen  (Quoted 
from  a professor  who  taught  Accounting  course) 

I saw  a training  video  that  a large  CPA  firm  created  for  their  staff,  it  had  three  windows,  it  had  a 
power  point  slide  with  the  main  points,  it  had  a window  with  the  actual  person  making  the 
presentation,  it  had  a window  of  that  person’s  script  scrolling,  so  you  could  actually  follow  what 
was  being  said,  and  compare  it  with  what  was  up  on  the  power  point  slide,  you  could  stop  it  and 
replay  it,  key  words  were  highlighted,  I mean,  it’s  extremely  engaging  in  the  sense  that  it’s 
much  closer  to  a classroom  experience  (Quoted  from  a professor  who  taught  Accounting  course) 

However,  accessibility  and  bandwidth  were  key  issues  that  concerned  the  instructors  in  their  adopting 
more  visually  rich  and  interactive  tools.  Some  instructors  noted  that: 

I think  the  idea  of  doing,  there’s  a lot  you  could  do  but  the  constraint  is  the  computers  at  the 
other  end  are  not  what  they  need  to  be.  I’d  love  to  get  on  a web  cam  and  have  a chat  room  with 
a web  cam  where  they  could  get  on  and  see  me  but  most  of  them  can’t  do  that  because  they 
don’t  have  the  software  or  the  hardware  to  do  it. 

I’ve  have  the  advantage  of  being  a professor  and  I’m  on  the  land  here  and  I have  high  speed  in 
my  office.  I have  DSL  technology  at  home.  I have  on  occasions  had  to  travel  and  been  in  a 
motel  room  and  had  to  connect  over  a phone  modem  and  it’s  thoroughly  frustrating  to  me  any 
more.  I think  students  who  still  have  that  technology  it’s  probably  thoroughly  frustrating  for 
them  also. 

Student  Surveys 

102  second-year  online  MBA  students  completed  the  survey  of  which  82.4%  were  males  and  17.6% 
were  females.  About  80%  of  those  responded  were  between  26  and  40  years  of  age.  90%  of  the  respondents 
took  more  than  seven  online  courses  in  the  program.  The  coefficient  of  reliability  was  performed  on  the 
instrument  to  check  for  its  reliability  and  the  Cronhach’s  alpha,  was  .91. 

The  Level  of  Instructional  and  Social  Presence  in  Online  Courses 

81.3%  of  the  respondents  reported  that  the  instructors  made  announcement  and  gave  feedback  to 
students  on  a regular  basis.  Receiving  such  regular  announcement  and  feedback  seems  to  strengthen  student 
feeling  of  being  part  of  a learning  community  (r=.46).  Receiving  the  regular  announcement  and  feedback  was 
also  moderately  correlated  with  the  overall  course  satisfaction  (r=.47)  and  negatively  correlated  with  the 
intention  of  dropping  out  of  the  class  (r=  -.51).  Male  students  tended  to  be  more  positive  than  female  students 
on  whether  the  online  instructors  foster  student  learning  (F=4.12,  pc. 05).  The  ANOVA  result  indicates  that 
there  is  a significant  difference  between  students  who  are  31-35  years  old  and  those  who  are  older  than  45  on 
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their  perception  of  whether  the  amount  of  orientation  they  received  on  the  electronic  course  management 
system  (ANGEL)  was  sufficient.  Students  at  age  31-35  said  that  they  have  had  sufficient  training  on  how  to  use 
ANGEL,  while  students  age  above  45responded  that  the  training  was  insufficient. 

What  Influences  Online  Students’  Engagement  in  their  Learning? 

About  90  percent  of  the  students  reported  that  they  were  deeply  engaged  in  their  learning  while  they 
took  the  online  courses  (M=4.17,  SD=.77).  The  correlation  coefficient  between  learner  engagement  and  the 
sense  of  a learning  community  was  fairly  high  (r=.62).  Student  engagement  was  also  moderately  correlated  with 
the  student’s  feeling  of  how  much  the  instructor’s  facilitation  fosters  learning  (r=.41).  The  student  learning 
engagement  was  also  positively  correlated  with  student  satisfaction  with  the  course  quality  (r=.65),  feeling 
about  learned  a lot  (r=.56),  and  intention  of  recommending  this  program  to  others  (r=,42).  In  contrast,  student 
engagement  was  negatively  correlated  with  the  student’s  intention  of  dropping  out  of  the  class  (r=  - .40).  The 
feeling  of  being  a part  of  a learning  community  was  positively  correlated  with  student  overall  course 
satisfaction  (r=.61),  feeling  of  learned  a lot  (r=.60),  and  their  intention  to  recommend  the  program  to  others 
(r=,54). 

Students’  Attitudes  Toward  Virtual  Teams  and  Online  Interactions 

Overall,  the  students  had  positive  attitudes  toward  teamwork  in  their  online  M BA  program  (M=4.27, 
SD=.72).  Approximately  93%  of  the  respondents  also  felt  that  sharing  information  and  giving  peer  feedback  in 
team  projects  contributed  to  student  learning  (M=4. 17,  SD=.63).  About  86%  of  the  students  surveyed  agreed  or 
strongly  agreed  that  working  in  groups  was  helpful  for  their  learning  (M=4.22,  SD=.91).  However,  only  49%  of 
the  students  agreed  with  the  statement  that  group  work  was  more  important  than  individual  work  when  learning 
online  (M=3.42,  SD=1.19,).  About  94%  of  the  respondents  think  that  interacting  with  other  students  or 
instructors  creates  a more  meaningful  learning  experience  (M=3.84,  SD=.84).  Also,  the  respondents  agreed  that 
interacting  with  other  students  motivates  them  to  explore  knowledge  more  deeply  (M=3.84,  SD=.84). 

Students’  Satisfaction  with  Online  Learning  and  Motivation 

About  93%  of  the  respondents  agreed  or  strongly  agreed  that  they  were  satisfied  with  the  quality  of 
online  courses  (M=4.27,  SD=0.72).  The  results  of  correlational  analysis  revealed  that  students’  satisfaction  with 
the  online  courses  was  positively  correlated  with  following  factors: 

• feeling  of  being  part  of  a learning  community  (r=.61) 

• engagement  in  learning  (r=.65) 

• instructor’s  use  of  various  instructional  techniques  to  foster  student’s  critical  and  reflective 
thinking  (r=.51) 

• technological  affordances  (r=.40) 

• effectiveness  of  instructor’s  facilitation  (r=.47) 

• feeling  of  being  part  of  a community  at  the  school  level  (r=.46) 

• feeling  of  having  learned  a lot  (r=.73) 

• academic  confidence  (r=.50) 

• prompt  feedback  from  the  instructor  (r=. 50). 

• informative  feedback  from  the  instructor  (r=.43) 

In  addition  to  their  high  level  of  satisfaction  with  their  online  courses,  the  students  appeared  to  be 
highly  motivated  to  persist  in  their  learning.  Only  8.8%  of  the  respondents  report  that  they  have  thought  about 
dropping  out  of  the  class  due  to  their  disappointment  with  the  course  design.  This  intention  of  dropping  out  of 
the  classes  was  negatively  correlated  with  learner  engagement  (r=-.40),  a sense  of  learning  community  (r=-.47), 
comfort  level  of  reading  messages  and  materials  online  (r=-.40),  and  helpfulness  of  the  instructor’s  facilitation 
(r=-.5 1).  About  96%  of  the  survey  respondents  also  intended  to  recommend  this  online  MBA  program  to  others, 
which  was  strongly  correlated  with  the  student’s  satisfaction  with  the  program  (r=.61). 

Conclusions  & Discussion 

The  findings  of  this  study  indicate  that  both  the  faculty  students  displayed  a high  level  of  satisfaction 
regarding  the  effectiveness  of  online  facilitation.  They  also  exhibited  positive  attitudes  toward  the  online 
learning  environment  in  general.  Additionally,  this  study  found  that  case-based  learning  was  being  used  in  a 
majority  of  online  MBA  courses  and  the  instructors  used  various  approaches  to  implement  case-based  learning 
in  their  online  courses.  Both  the  faculty  and  the  students  perceived  case-based  learning  as  an  effective  way  to 
teach  an  online  MBA  course.  Also,  most  of  the  courses  incorporated  a team-based  learning  approach  students 
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interacted  with  one  another  also  with  the  instructor.  In  terms  of  student  motivation,  the  students  indicated  a high 
level  of  motivation  for  completing  their  program,  which  is  due,  in  part,  to  their  goal  to  obtain  a degree  as  well  as 
to  the  flexibility  and  convenience  of  an  online  program. 

Both  instructors  and  students  of  online  MBA  courses  perceived  that  interactions  among  the  instructor  and 
the  students  as  well  as  among  students  were  not  sufficient  and  the  sense  of  community  among  the  online 
students  was  not  strong.  Such  issues  raised  from  the  findings  of  this  study  indicate  that  the  faculty  and  students 
both  need  more  guidance  and  support  technologically  and  pedagogically  to  create  a more  engaging  and 
meaningful  online  learning  environment. 

The  results  of  this  study  on  the  effectiveness  and  issues  of  online  facilitation  will  be  useful  for  distance 
educators  and  policy  makers  of  online  programs  who  are  conducting  strategic  planning,  making  educational 
policies,  or  refining  practices  for  providing  more  satisfactory  educational  experiences  in  online  learning 
environments.  This  study  will  also  provide  a set  of  assessment  instruments,  models,  and  guides  for  those 
researching  similar  programs.  Yet,  this  paper  reports  preliminary  results  of  an  on-going  research  study,  and  the 
analysis  of  faculty  survey  and  student  survey  data  are  yet  to  be  done.  Further  analyses  of  data  are  expected  to 
provide  more  in-depth  understanding  and  implications  for  theory  and  practice  from  the  findings  of  this  study. 
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Impediments  to  Faculty  Engaging  in  Web-based  Instruction:  Clarification  of 

Governing  Policies 

Wanjira  Kinuthia 
Georgia  State  University 

Abstract 

An  exploratory  study  was  conducted  with  faculty  and  administrators  at  Historically  Black  College  and 
Universities  (HBCUs)  to  determine  the  level  of  reported  knowledge  and  experience  with  policies  that  govern 
Web-based  instruction  (WBI).  Result  indicated  that  faculty  had  little  experience  with  the  policies  and  many  had 
not  participated  in  policy  development.  Lack  of  communication  of  the  policies  was  also  reported  to  be  a barrier 
to  participation.  Recommendations  on  policy  information  adoption  and  dissemination  were  made  based  on  the 
findings  of  the  study. 


Purpose  of  the  Study 

Higher  education  is  in  continual  transition  and  one  cannot  underestimate  the  influence  of  technology 
on  everyday  life  of  learners  and  educators.  Web-based  Instruction  (WBI)  is  among  the  forces  influencing  higher 
learning  and  instruction  and  Historically  Black  Colleges  and  Universities  (HBCUs)  have  not  been  left  out. 
However,  while  technology  offers  new  options,  there  is  also  the  danger  of  losing  some  of  the  important 
attributes  of  higher  education.  These  attributes  include  the  commitment  to  providing  the  less  advantaged  with  an 
opportunity  for  education,  education’s  tradition  of  addressing  student  and  societal  needs,  and  the  emphasis  on 
learning  and  scholarship.  There  is  a technology  and  information  gap  that  is  evident  in  the  Black  community  and 
this  his  gap  has  kept  the  poor,  rural,  and  minority  populations  from  participating  and  benefiting  from  the 
information  technology  revolution  and  this  trend  is  inevitably  carried  on  to  the  Black  college  environment 
(Hamilton,  2001). 

While  there  are  numerous  studies  examining  older  methods  of  distance  education,  few  comprehensive 
empirical  studies  have  provided  evidence  of  the  effectiveness  of  WBI  in  HBCUs.  Little  research  has  been 
conducted  in  the  area  of  WBI  focusing  specifically  on  input  from  faculty.  The  purpose  of  this  study  was  to 
explore  the  factors  that  influence  faculty  participation  in  WBI  at  HBCUs.  This  exploratory  study  was  conducted 
to  determine  the  factors  that  influence  faculty  participation  in  WBI  at  seven  HBCUs,  including  determining  the 
level  of  reported  knowledge  and  experience  with  policies  that  govern  WBI.  In  this  mixed-design  study,  data 
used  to  investigate  the  research  questions  came  from  responses  from  one  hundred  and  forty-nine  faculty  and 
administrators  (deans  and  departments  chairs)  at  seven  public  HBCUs  that  offer  online  curricula.  An  online 
questionnaire  was  designed  to  collect  data  using  a four-point  scale,  open-ended  items,  and  follow-up  interviews. 

Background 

Lack  of  clarification  of  WBI  policies  is  a barrier  to  participation  for  many  faculty.  These  polices  can 
be  categorized  into  several  areas:  (1)  Academic  policies  refer  to  quality,  accreditation,  grading,  program 
evaluation,  admissions,  credentialing,  mission  compliance  and  curriculum  review;  (2)  Fiscal,  geographical,  and 
governance  policies  cover  fees,  in -state  and  out-of-state  relationships,  consortia  agreements,  and  contracts  with 
collaborating  agreements;  (3)  Faculty-related  Web-based  policies  address  compensation,  workload,  design  and 
development,  incentives,  staff  development,  support,  evaluation,  and  intellectual  freedom  issues;  (4)  Legal 
policies  refer  to  intellectual  property  agreements,  copyright,  and  faculty/student/institutional  liability;  (5) 
Student-related  policies  address  support,  access,  advising,  training,  financial  aid,  assessment,  access  to 
resources,  equipment  requirements,  and  privacy;  (6)  Technical  polices  define  reliability,  connectivity,  technical 
support,  hardware/software  access;  and  (7)  Philosophical  policies  are  developed  to  define  a clear  understanding 
of  approach,  faculty  autonomy,  organizational  values,  and  missions,  enhanced  public  access,  organization, 
governance,  partnerships,  and  financial  support  are  all  themes  that  should  be  addressed  in  the  discussion  on 
policy  and  Web-based  instruction  (Gellman-Danley,  1997,  Hickman,  1999;  Noble,  1998;  Simonson,  2002). 

Kinyanjui  (1998)  states  that  distance  education  is  often  criticized  because  governing  policies  are  often 
not  coordinated  with  provision  of  resources,  development  of  supporting  infrastructures,  and  training  of  users  of 
distance  education.  He  also  indicates  that  distance  education  is  introduced  without  adequate  understanding  of 
the  organizational  culture  and  context,  and  the  political,  physical,  economic,  social,  and  technological 


476 


environment.  Kinyanjui  further  observes  that  distance  education  is  sometimes  introduced  hastily  or  arbitrarily  in 
a top-down  manner.  Inadvertently,  decisions  should  be  made  as  to  whether  a top-down  or  a bottom-up  approach 
should  be  used  to  integrate  technology.  The  top-down  approach  assumes  that  formulating  goals,  organizational 
structures,  management  approaches,  implementing  technological  advancements  should  bring  about  change 
(Surry  & Farquahr,  1997).  On  the  other  hand  the  bottom-up  approach  is  one  that  should  facilitate  change  from 
the  point  of  view  of  middle -level  administrators,  faculty,  and  learners,  who  work  directly  with  the  technology 
(Fitz  & Haplin,  1994). 

Understanding  the  fundamental  characteristics  that  shape  HBCUs  serves  as  a framework  of  analysis 
for  equality  and  access  to  higher  education.  HBCUs  are  postsecondary  academic  institutions  that  were  founded 
prior  to  1964,  and  whose  educational  mission  has  historically  been  the  education  of  Black  Americans. 
Predominantly  Black  institutions  of  higher  education  are  classified  as  non-HBCUs  that  serve  a majority  of 
Black  students.  Predominantly  black  colleges  and  universities  are  institutions  that  were  not  founded  primarily 
for  African  Americans  but  have  more  than  50  percent  black  student  enrollment.  There  are  103  HBCUs  and  over 
50  predominantly  Black  colleges  and  universities  located  in  twenty  states,  the  District  of  Columbia,  and  the 
Virgin  Islands  (Brown  & Davis,  2001).  These  institutions  include  accredited  two-  and  four-year  schools,  and 
graduate  and  professional  institutions.  Forty-nine  percent  are  public  and  fifty-one  percent  are  private,  and  they 
generally  face  the  same  issues  as  HBCUs  (Brown  & Davis,  2001;  Brown  & Hendrickson,  1997;  Evans,  et  ah, 
2000). 

HBCUs  encounter  challenges  pertaining  to  improving  the  technology  infrastructure,  training  faculty 
for  online  teaching,  and  developing  online  content.  These  challenges  are  further  complicated  by  the  fact  that 
HBCUs  generally  have  fewer  monetary  resources,  charge  their  learners  less,  and  have  to  take  their  historical 
mission  of  cultivating  a supportive  atmosphere  for  their  learners  into  account  (Hamilton,  2001,  Brown  & Davis, 
2001).  The  cost  of  technology  is  a barrier  that  has  been  an  area  of  concern  for  some  HBCUs,  especially  those 
who  have  limited  financial  resources.  To  address  these  issues,  many  institutions  seek  supplemental  funding  and 
many  have  formed  consortia  that  create  Web-based  courses  and  programs  Hamilton,  2001). 

Design  of  the  Study 

This  exploratory  study  was  conducted  using  quantitative  and  qualitative  data  collection  and  analysis 
methods.  Data  used  to  investigate  the  research  questions  came  from  responses  to  an  online  questionnaire 
designed  to  collect  quantitative  data  using  a four-point  scale.  Open-ended  questions  were  included  at  the  end  of 
each  section  and  follow-up  interviews  were  conducted  with  faculty  and  administrators  via  e-mail.  Documents 
and  official  institution  Web  pages  were  also  reviewed  for  addition  information. 

Institutions  selected  for  participation  in  the  study  are  all  public  four-year  institutions  located  in  the 
southern  United  States.  The  seven  HBCUs  were  selected  based  on  the  following  criteria.  First,  institutions  that 
offer  online  curricula  were  selected.  Second,  all  the  HBCUs  use  Course  Management  Systems  such  as  WebCT 
© and  Blackboard  © or  have  institution  specific  software  for  WBI.  Third,  they  provide  faculty  support  and 
development  for  WBI  in  various  forms,  such  as  workshops  and  seminars. 

The  participants  in  the  study  were  faculty  and  administrators  (deans  and  department  chairs).  E-mail 
with  a link  to  the  Website  where  the  survey  instrument  was  located  was  sent  to  1 125  faculty  and  administrators 
inviting  them  to  respond  to  the  survey.  There  were  152  (12.4%)  valid  responses  with  female  participants 
accounting  for  54%  of  the  participants,  while  males  accounted  for  46%  of  the  responses.  Overall,  61%  of  the 
participants  were  Black  or  African  American,  32.9%  were  White,  3.3%  were  Asian,  and  1.3%  were  Native 
Hawaiians.  Approximately  68%  of  the  participants  indicated  their  primary  activity  was  teaching  and  23% 
identified  themselves  as  administrators.  A majority  of  the  participants  (68%)  were  associate  (32.2%)  and 
assistant  (31.6%)  professors.  Professors  made  up  21.7%  of  the  total  participants  and  instructors  made  up  10.5% 
of  the  responses.  Four  follow-up  interviews  were  conducted  with  four  administrators  and  faculty  at  participating 
institutions. 


Findings  of  the  Study 

Quantitative  and  qualitative  data  were  collected  in  this  study.  Participants  responded  to  four  questions 
about  their  levels  of  experience  with  polices  that  govern  WBI  at  their  institutions.  They  rated  each  of  the 
questions  using  the  following  scale;  ( 1)  No  Experience;  (2)  A Little  Experience;  (3)  Some  Experience;  and  (4) 
High  Experience.  The  results  are  presented  in  Table  1.  Out  of  149  responses,  59.7%  of  the  faculty  indicated  that 
they  had  some  or  high  amount  of  knowledge  about  general  WBI  polices  at  their  institutions.  However, 
participants  typically  reported  that  they  had  not  participated  in  the  actual  development  of  the  policies,  and  only 
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6.7%  indicted  that  their  level  of  participation  in  policy  development  was  high.  One  participant  acknowledged 
familiarity  with  the  policies  at  her  institution  and  stated: 

I am  somewhat  familiar  with  policies  regarding  Web-based  courses;  however  I have  yet  to 
participate  in  the  development  of  any  of  these  policies  here.  We  are  currently  developing 
Web-based  courses  for  our  electronic  classroom  (this  Fall).  However,  we  are  bounded  by  the 
University  policies  governing  WBI. 

A participant  at  a different  university  indicate  he  was  also  aware  of  the  institutions  WBI  polices  but 
stated  that  he  lacked  time  to  be  more  active  in  the  policy  development  process.  He  noted  that  a Technology 
Committee  had  been  active  for  several  years  but  due  to  other  commitments  he  was  not  part  of  the  decision- 
making processes.  Another  participant  who  indicated  she  was  aware  of  the  policies  at  her  institution  emphasized 
that  “to  know  the  policies  is  not  the  same  as  participating  in  the  development  of  the  polices.” 

Looking  at  one  specific  area  of  WBI  polices,  72%  of  the  participants  indicated  that  they  had  little  or  no 
knowledge  of  intellectual  property  rights  policies  at  their  institution  and  only  2.7%  had  high  experience  in 
helping  develop  these  intellectual  property  policies.  Participants  indicated  their  institutions  should  have  clear 
policies  on  intellectual  property  rights  to  encourage  faculty  to  participate  in  WBI.  One  person  expressed  this 
concern  as  follows: 

WBI  was  covered  in  an  agreement.  With  the  destruction  of  the  governing  organization  there 
is  now  a question  about  the  bargained  agreement  and  the  whole  issue  of  intellectual  property 
rights  at  my  University. 

In  addition  to  lack  of  experience  with  policies,  participants  indicated  that  their  institutions  did  not 
communicate  the  policies  clearly  but  that  they  would  be  willing  to  develop  Web-based  courses  with 
clarification  of  the  policies.  One  faculty  stated: 

I have  a reluctance  to  participate  in  WBI  for  various  reasons.  I am  not  sure  of  the  intellectual 
property  policies  at  all.  I think  there  are  some;  however,  no  one  has  been  able  to  clearly 
articulate  them  to  me. 

Several  participants  indicted  that  they  had  attended  one  or  two  seminars  on  intellectual  property  rights 
and  were  aware  of  guidelines.  Another  participant  indicated  that  WBI  was  new  at  his  university  and  that  he  was 
aware  that  the  distance  education  department  was  in  the  process  of  developing  policies  in  accordance  with  state 
guidelines. 

Table  1 Experience  with  and  Knowledge  of  WBI  Polices 


No  Exp 

A Little  Exp 

Some  Exp 

High  Exp 

Mean 

Experience  with 
Policies 

n 

% 

n 

% 

n 

% 

n 

% 

N 

SD 

M 

Rank 

Knowledge  of 
general  policies 

26 

17.4 

34 

22.8 

73 

49.0 

16 

10.7 

150 

.90 

2.53 

1 

Knowledge  of 
Intellectual 
Property  Rights 

49 

32.7 

37 

24.7 

54 

36.0 

10 

6.7 

151 

.96 

2.17 

2 

Participated  in 
Policy  formation 

81 

54.0 

27 

18.0 

36 

24.0 

6 

4.0 

151 

.94 

1.79 

3 

Participated  in 

97 

64.7 

30 

20.0 

19 

12.7 

4 

2.7 

151 

.82 

1.54 

4 

Intellectual 
Property  Rights 
Policy  Formation 

Note:  1 = No  Experience;  2 = A Little  Experience;  3 = Some  Experience;  4 = High  Experience 


The  common  threads  of  HBCUs  discussed  by  Brown  and  Davis  (2001)  are  social  organization  and 
Black  cultural  tradition  which  seeks  to  provide  leadership  for  the  Black  community.  HBCUs  play  a role  in 
interpreting  social,  political,  and  economic  dynamics  and  addressing  overarching  issues  between  the  minority 


478 


and  majority  population  groups.  HBCUs  also  act  as  agents  for  specialized  research,  institutional  training,  and 
information  dissemination  for  Black  and  other  minority  communities.  Overall,  faculty  reported  that  they  had 
little  experience  with  general  and  intellectual  property  policies.  Typically,  they  reported  that  they  had  not 
participated  in  either  formation  of  general  or  intellectual  rights  policies.  Lack  of  communication  on  the  policies 
was  also  reported  to  be  a barrier  to  participation  in  WB1  by  five  participants. 

Discussion 

Faculty  in  this  study  generally  reported  that  they  had  little  experience  and  knowledge  of  WB1. 
Participants  reported  they  had  not  taken  part  in  the  development  of  WB1  policies,  and  many  were  not  aware  of 
efforts  by  their  administration  to  communicate  related  information.  Other  participants  indicated  that  while  they 
were  willing  to  participate  in  committees  that  addressed  policy  issues,  they  felt  they  had  an  obligation  to  attend 
to  what  they  felt  were  far  more  pressing  issues  such  as  teaching,  conducting  research,  publishing,  and  service. 
This  lack  of  participation  is  contrary  to  recommendations  by  Czubaj  (2001)  who  states  that  an  area  of  particular 
significance  is  increasing  the  awareness  and  knowledge  among  faculty,  staff,  and  students  about  educational 
technologies  and  methods  that  have  the  potential  to  enhance  the  outreach  mission  of  the  individual  institutions. 

Legal  issues  that  included  intellectual  property  rights  and  institutional,  faculty,  and  student  liability 
were  an  on-going  concern  with  faculty.  Unclear  guidelines  on  intellectual  property  rights  often  serve  as  a barrier 
to  participation  in  WB1  for  many  faculty  (Hill,  1997;  West,  1999).  Potential  participants  are  apprehensive  about 
developing  coursework  for  the  WBI  environment  until  they  have  clear  knowledge  of  who  owns  the  material  as 
it  is  often  unclear  as  to  who  owns  the  rights  to  the  instructional  material. 

With  the  exponential  growth  of  the  Internet  and  the  Web,  the  challenge  that  remains  is  keeping  the 
practices  and  guidelines  current  and  accurate.  While  each  HBCU  has  its  own  administrative  approach,  it  is 
important  for  administrators  and  faculty  to  communicate  with  each  other  and  to  work  together  in  policy 
development.  Based  on  individual  institutional  missions,  Czubaj  (2001)  suggests  forming  program  evaluation 
advisory  committees  with  representation  from  students,  faculty,  staff,  and  administration.  Commitment  should 
encompass  academic  affairs,  information  technology  services,  library  services,  and  partnerships  and  liaisons 
who  work  with  the  programs. 

Continuous  development  of  leadership  skills,  development  of  innovative  solutions  to  fulfill  HBCU 
missions,  serve  students,  and  involvement  of  faculty  in  decision-making  processes,  are  all  important  elements. 
Collins  (2001)  and  Lape  and  Hart  (1997)  note  that  many  tenets  of  educational  technology  integration  are 
emerging  thus  the  need  for  committed  leadership  in  bringing  educational  technology  into  instruction.  Hence, 
before  institutional  policies  can  be  changed,  each  individual  institution  should  determine  what  the 
administration  knows  about  WBI  and  the  importance  that  is  placed  on  the  policies. 

To  address  policies  for  successful  WBI  integration,  several  approaches  are  recommended.  First, 
policies  should  be  integrated  gradually  and  seamlessly  to  incorporate  the  concept  of  distant  delivery  of 
instruction.  Second,  learners  should  be  defined  by  their  enrollment  in  a course,  not  by  whether  they  are  distant 
or  on  campus.  With  the  mission  of  many  HBCUs  being  to  provide  a nurturing  learning  environment,  designing 
a learner-friendly  environment  is  important.  Third,  initially  policies  should  be  separate  from  existing  policies. 
Ultimately,  policies  can  be  integrated  to  indicate  that  WBI  is  a regular  component  of  instructional  delivery,  as 
faculty  become  more  proficient  with  the  technology. 

When  making  recommendations  for  WBI  policies,  several  questions  should  be  addressed  before 
faculty  settle  on  developing  and  teaching  Web-based  courses.  It  is  recommended  that  each  HBCU  respond  to 
concerns,  most  of  which  where  raised  by  participants  in  this  study:  Do  faculty  have  portability  rights  to  take  the 
material  with  them  when  they  leave?  In  the  event  that  another  faculty  member  teaches  the  course,  does  the 
developer  receive  compensation?  Should  copyright  be  in  the  name  of  the  developer  or  the  school?  If  these 
questions  cannot  be  clearly  answered,  it  is  unlikely  that  faculty  will  be  willing  to  participate  in  WBI. 

Based  on  the  above  questions,  the  following  is  recommended:  (1)  Copyright  ownership  policies  written 
to  allow  faculty  to  reasonably  have  latitude  with  their  own  work;  (2)  As  with  classroom-based  courses,  the 
extent  that  institutions  have  the  authority  to  determine,  suggest,  or  decide  use  of  Web-based  course  material 
should  be  clearly  addressed;  (3)  It  is  to  the  advantage  of  the  HBCU  and  the  faculty  to  define  each  participating 
member's  ownership  rights.  Hence,  all  parties  should  know  who  owns  the  final  product;  (4)  Compensation  and 
workload,  design  and  development  incentives,  support,  and  promotion  and  tenure  should  also  be  taken  into 
account.  When  faculty  have  leverage  with  their  work  and  they  are  part  of  the  decision  making  process  they  will 
be  more  likely  to  take  initiative  and  ownership  of  the  Web-based  course  development  process  and  end-product 
(Gellman-Danley,  1997). 

The  following  recommendations  are  also  made  based  on  the  concerns  raised  by  participants  regarding 
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the  development  and  communication  of  WBI  policies:  (1)  Building  solid  and  diverse  relationships  across 
academic  units  to  assist  in  policy  issues;  (2)  Developing  university-wide  committees  to  review  policies  and 
communicates  guidelines;  (3)  Publishing  policies  in  institution  documents,  on  relevant  parts  of  the  university 
Web  site,  and  other  visible  locations  in  the  college  community;  (4)  Developing  institutional  expertise  in  policy 
issues  thus  developing  collegiality  and  distribute  appropriate  training;  and  (5)  Using  the  guidelines  and  policies 
to  balance  the  interests  of  intellectual  property  rights  of  the  faculty  and  institution. 

Noblitt  (1997)  states  that  the  key  to  collaborative  decision-making,  when  incorporating  technology  into 
education  is  for  both  bottom-up  and  top-down  administrators  to  ask  the  following  questions  when  reviewing 
WBI  guidelines:  Are  there  mission-critical  problems  that  are  unresolved  under  the  current  practices?  Do  the 
problems  affect  a majority  of  the  students?  Does  technology  provide  any  real  value  educationally?  Can  the 
project  be  implemented  with  existing  resources?  Is  technology  a solution  for  these  problems?  If  the  answer  is 
no,  then  perhaps  technology  is  not  the  solution. 


Conclusion 

The  area  of  integration  of  technology  in  education  is  a continuous  effort  that  revolves  around  looking 
for  factors  and  practices  that  can  be  applied  to  encourage  faculty  to  integrate  technology  in  their  areas  of 
teaching.  Since  the  HBCUs  selected  for  the  study  offered  online  curricula,  this  study  focused  on  investigating 
the  factors  that  influence  faculty  participation  in  WBI.  Outcomes  of  this  study  indicated  that  faculty  had  little  or 
no  experience  with  Web-course  policies  and  faculty  were  not  actively  involved  in  corresponding  policy 
development. 

The  question  of  suitability  of  WBI  for  both  their  institutions  and  their  learners  is  an  area  of  concern  for 
many.  Is  WBI  policy  formation  an  initiative  of  administrations  or  voluntary  action  on  the  part  of  the  faculty?  If 
it  is  voluntary,  how  much  input  from  faculty  was  taken  into  account?  Was  WBI  policy  formation  a top-down  or 
bottom-up  approach?  When  planning,  implementing,  and  maintaining  WBI,  governing  policies  regarding  the 
program  must  be  carefully  developed.  Planning  should  include  needs  assessment,  policy  barriers,  and 
evaluation.  Because  technology  is  constantly  changing,  the  policies  also  require  regular  revision  and  updating. 
As  technology  continues  to  evolve,  the  policies  governing  WBI  will  become  more  complex  and  accumulative. 
Inadvertently,  the  policies  will  require  continuous  review  to  remain  current  and  valid. 
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Abstract 

This  paper  focuses  on  the  research  competencies  of  students  in  Instructional  Design  and  Technology 
(IDT)  programs.  First,  we  describe  how  research  is  being  taught  to  students  enrolled  in  severed  leading  IDT 
programs.  Next,  we  present  findings  from  a survey  administered  to  determine  what  knowledge  and  skills  are 
being  taught  to  IDT  graduate  students.  Implications  for  the  design  of  courses  that  focus  on  research  are 
discussed. 


Introduction 

The  competencies  of  professionals  in  the  Instructional  Design  and  Technology  (IDT)  field  continue  to 
receive  attention.  In  recent  years,  books,  articles,  and  conference  presentations  have  focused  on  the 
competencies  of  instructional  designers  (Richey,  Fields,  & Foxon,  2001),  training  managers  (Foxon,  Richey, 
Roberts,  & Spannaus,  2003),  instructors  in  face-to-face  and  online  settings  (Goodyear,  et.  al.  2001;  Klein, 
Spector,  Grabowski,  & de  la  Teja,  2004;  Spector  & de  la  Teja,  2001),  and  performance  technologists  (Fox  & 
Klein,  2003). 

The  current  study  continues  a line  of  research  conducted  to  identify  the  knowledge  and  skills  that  students 
in  graduate  IDT  programs  should  obtain.  An  earlier  study  examined  the  optimal  content  and  delivery  method 
for  a “foundations’'  course  in  Educational  Technology  (Klein,  Brinkerhoff,  & Koroghlanian,  2003).  Another 
study  focused  on  the  skills  that  IDT  students  should  learn  related  to  performance  improvement  (Klein  & Fox, 
2003).  The  purpose  of  the  current  work  is  to  answer  the  following  two  questions: 

• How  is  research  being  taught  to  IDT  graduate  students? 

• What  research  methods,  processes,  and  issues  are  being  taught  to  IDT  graduate  students? 

Method  - Phase  1 

During  this  phase  of  the  study,  we  obtained  and  analyzed  the  syllabi  from  research  courses  offered  at 
several  leading  IDT  programs  to  determine  how  re  search  is  being  taught  to  graduate  students  in  the  field. 

Participants 

Our  sample  consisted  twelve  IDT  programs  listed  in  the  2002  Educational  Media  and  Technology 
Yearbook  (EMTY).  We  only  sampled  programs  that  EMTY  listed  as  offering  a Ph.D.  degree  in  Instructional 
Technology  or  Instructional  Design  and  Development.  The  following  institutions  were  included  in  our  initial 
sample  - Arizona  State  University,  Florida  State  University,  Indiana  University,  Pennsylvania  State  University, 
Purdue  University,  Syracuse  University,  University  of  Georgia,  University  of  Memphis,  University  of  Northern 
Colorado,  Utah  State  University,  and  Wayne  State  University. 

Procedures 

We  conducted  a web  search  of  each  IDT  program  at  the  institutions  listed  above  to  determine  their 
research  course  offerings.  Next,  we  obtained  the  syllabi  for  research  courses  offered  by  each  program.  Some  of 
the  course  syllabi  were  found  on  the  web,  while  others  were  obtained  by  making  direct  contact  with  individual 
faculty  members  at  each  program.  Syllabi  for  research  courses  offered  at  10  or  the  12  universities  were  obtained 
and  include  in  our  analysis. 

We  then  conducted  a content  analysis  of  the  research  course  syllabi.  Each  syllabus  was  examined  for 
the  following:  course  title,  credit  hours,  objectives,  textbooks  and  other  readings  used,  topics  covered, 
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instructional  activities,  projects,  and  assignments. 


Results 

Several  trends  were  identified  by  the  content  analysis.  These  trends  are  reported  below. 

• Most  of  the  IDT  programs  in  the  sample  offer  their  own  research  courses.  Several  programs  offer  more 
than  one  research  course.  A few  programs  rely  on  others  in  their  college  to  offer  these  courses. 

• Most  of  the  courses  examined  focus  on  doing  research.  Many  included  requirements  such  as  planning 
and  conducting  a research  study.  Some  related  to  planning  a dissertation  study.  Others  focused  on 
specific  phases  such  as  forming  a research  problem  and  conducting  a literature  search. 

• A range  of  quantitative  and  qualitative  methodologies  is  taught  in  IDT  research  courses. 

• A wide  variety  of  textbooks  are  used  in  IDT  research  courses.  While  several  textbooks  on  research 
methodology  have  been  adopted,  preliminary  analyses  did  not  suggest  any  book  as  being  most  widely 
used.  Several  courses  have  students  read  excerpts  from  different  textbooks. 

• A few  courses  require  students  to  read,  interpret,  and  analyze  primary  source  documents  such  as 
published  research  articles. 


Method  - Phase  2 

During  this  phase  of  the  study,  we  developed  and  administered  an  online  survey  to  determine  what 
research  methods,  processes,  and  issues  are  taught  to  IDT  graduate  students. 

Participants 

We  sent  a request  for  participation  to  two  division  listservs  owned  by  the  Association  of  Educational 
Communications  and  Technology  (Design  & Development  Division  and  Research  & Theory  Division)  and  to  a 
listserv  owned  by  the  Professors  of  Instructional  Design  and  Technology  (PIDT).  In  addition,  a request  was  sent 
to  consulting  editors  of  Educational  Technology  Research  and  Development  and  to  a sample  of  individuals 
listed  on  the  website,  Who’s  Who  in  Instructional  Design  and  Technology. 

Our  request  led  to  50  graduate  students  and  50  faculty  members  who  completed  the  survey  (N  = 100). 
Respondents  represented  over  35  IDT  programs  mostly  located  in  the  United  States.  Three  programs  were 
located  outside  the  U.S.  (Australia,  China,  the  Middle  East).  However,  not  all  respondents  listed  their  program 
affiliation  when  given  the  option. 

Survey  Instrument 

The  first  section  of  the  survey  provided  a lis  t of  15  different  research  methods  (see  Table  1)  and  asked 
respondents  to  rate  the  degree  to  which  students  learn  about  each  method  using  the  following  scale: 

0 = This  research  method  is  not  covered  in  our  curriculum. 

1 = Students  are  expected  to  acquire  knowledge  related  to  this  method. 

2 = Students  are  expected  to  acquire  skills  related  to  this  method. 

3 = Students  are  expected  to  acquire  both  knowledge  & skills  related  to  this  method. 

The  second  section  of  the  survey  examined  the  degree  to  which  students  receive  formal  instruction  on  how 
to  conduct  a research  study  (see  Table  2).  Respondents  used  the  following  scale  to  rate  ten  research  processes: 

0 = Students  are  not  taught  how  to  do  this  research  process. 

1 = Students  are  taught  how  to  do  this  process  but  never  required  to  do  it. 

2 = Students  are  taught  how  to  do  this  process  and  do  it  for  the  first  time  during  their  thesis  or 

dissertation. 

3 = Students  are  taught  this  process  and  do  it  before  conducting  their  thesis  or  dissertation. 

The  third  section  of  the  survey  focused  on  five  issues  related  to  conducting  research  (see  Table  3)  and 
asked  respondents  to  rate  the  degree  to  which  students  acquire  knowledge  and  skills  related  to  each  issue  using 
the  same  scale  presented  in  section  one. 

The  fourth  section  of  the  survey  asked  respondents  to  identify  whether  they  were  a student  or  a faculty 
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member.  Faculty  were  asked  if  they  taught  graduate-level  research  courses  and  if  they  supervised  research 
theses  and/or  dissertations.  An  optional  item  asked  respondents  to  provide  the  name  of  their  university  and 
program. 


Results 

Table  1 provides  the  rankings  and  mean  scores  for  the  15  research  methods  included  in  the  survey. 
These  data  show  that  five  research  methods  (evaluation,  qualitative,  survey,  experimental,  & 
design/development  research)  received  an  overall  rating  between  2.06  - 2.32  indicating  that  most  IDT  students 
are  expected  to  acquire  skills  related  to  these  methods.  Four  other  research  methods  (quasi-experimental,  case 
study,  mixed- methods,  and  descriptive  research)  received  an  overall  rating  of  1.76  - 1.94  suggesting  that 
students  in  many  programs  are  expected  to  obtain  skills  related  to  these  methods. 

The  research  method  data  were  analyzed  to  determine  if  any  differences  existed  between  students  and 
faculty  members.  A significant  difference  was  found  for  quasi-experimental  methods  [F  (1,98)  = 7.86,  p < .01] 
and  action  research  [F  (1,98)  = 14.29,  p < .001].  In  both  cases,  faculty  members  rated  these  methods  higher 
than  students. 

Table  2 shows  that  nine  of  the  ten  research  processes  included  in  the  survey  received  an  overall  rating 
between  2.58  - 2.83  indicating  that  most  IDT  students  are  taught  these  processes  and  do  them  before  conducting 
their  thesis  or  dissertation.  A significant  difference  was  found  for  analyzing  & interpreting  research  data  [F 
(1,98)  = 7.76,  p < .01],  writing  research  reports  [F  (1,98)  = 10.56,  p < .01],  sampling  participants  [F  (1,98)  = 
8.46,  p < .01],  and  developing  data  collection  instruments  [F  (1,98)  = 12.31,  p < .001].  In  all  cases,  faculty 
members  rated  these  research  processes  higher  than  students. 

Table  3 provides  the  rankings  and  mean  scores  for  the  five  research  issues  included  in  the  survey. 

These  data  reveal  that  two  issues  (electronic  searches  & databases  and  analyzing  research  studies)  received  an 
overall  rating  between  2.43  and  2.36  respectively  indicating  that  most  IDT  students  are  expected  to  acquire 
skills  related  to  these  issues.  The  three  other  issues  (ethics,  trends  in  IDT  research,  and  professional  contexts  for 
research)  received  an  overall  rating  above  1.50  suggesting  that  many  IDT  students  are  expected  to  acquire  some 
knowledge  about  these  topics. 


Table  1 - Rankings  & Means  for  Research  Methods 


Ranking 

Research  Method 

Faculty 

Students 

Total 

1 

Evaluation 

2.54 

2.10 

2.32 

2 

Qualitative 

2.34 

1.96 

2.15 

3 

Survey  research 

2.26 

2.02 

2.14 

4 

Experimental 

2.18 

1.98 

2.08 

5 

Design/Development  research 

2.10 

2.02 

2.06 

6 

Quasi-experimental* 

2.22 

1.66 

1.94 

7 

Case  study 

2.04 

1.58 

1.81 

8 

Mixed-methods  research 

1.88 

1.72 

1.80 

9 

Descriptive  research 

2.00 

1.52 

1.76 

10 

Action  research* 

1.72 

.92 

1.32 

11 

Ethnography 

1.32 

1.08 

1.20 

12 

Meta-analysis 

1.18 

1.10 

1.14 

13 

Narrative  research 

.88 

.92 

.90 

14 

Historical  research 

.66 

1.02 

.84 

15 

Philosophical  inquiry 

.64 

.74 

.69 

* A significant  difference  was  found  between  faculty  and  students  (p  < .01). 
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Table  2 - Rankings  & Means  for  Research  Processes 


Ranking 

Research  Processes 

Faculty 

Students 

Total 

1 

Specifying  research  questions 

2.94 

2.72 

2.83 

2 

Conducting  a literature  review 

2.94 

2.68 

2.81 

3 

Selecting  a research  topic 

2.86 

2.74 

2.80 

4 

Identifying  research  variables 

2.90 

2.64 

2.77 

5 

Constructing  research  hypothesis 

2.80 

2.66 

2.73 

6 

Analyzing  & interpreting  research  data* 

2.90 

2.54 

2.72 

7 

Writing  research  reports* 

2.90 

2.46 

2.68 

8 

Selecting  a research  design 

2.76 

2.52 

2.64 

9 

Sampling  participants* 

2.78 

2.38 

2.58 

10 

Developing  data  collection  instruments* 

2.74 

2.14 

2.44 

* A significant  difference  was  found  between  faculty  and  students  ( p < .01). 

Table  3 - Rankings  & Means  for  Research  Issues 

Ranking 

Research  Issue 

Faculty 

Students 

Total 

1 

Electronic  searches  and  databases 

2.46 

2.40 

2.43 

2 

Analyzing  research  studies 

2.60 

2.12 

2.36 

3 

Ethics 

1.74 

1.78 

1.76 

4 

Trends  in  IDT  research 

1.78 

1.64 

1.71 

5 

Professional  contexts  of  research 

1.50 

1.56 

1.53 

Discussion 

The  purpose  of  this  study  was  to  investigate  the  research  competencies  of  graduate  students  in  IDT 
programs  by  examining  the  skills  and  knowledge  students  acquire  and  how  they  attain  them.  The  study  was 
conducted  in  two  phases.  In  Phase  1,  we  examined  the  syllabi  from  research  courses  offered  at  10  leading  IDT 
programs  to  determine  how  research  is  being  taught  to  graduate  students  in  the  field.  In  Phase  2,  we  conducted 
an  online  survey  to  determine  what  research  methods,  processes,  and  issues  are  taught  to  IDT  graduate  students. 

Results  from  both  phases  of  the  study  revealed  that  a range  of  quantitative  and  qualitative 
methodologies  is  being  taught  to  students  enrolled  in  IDT  research  courses.  An  interesting  finding  to  emerge 
was  that  IDT  faculty  and  students  rated  evaluation,  qualitative,  and  survey  methods  somewhat  higher  than 
experimental  and  quasi-experimental  methods.  This  finding  can  be  explained  by  the  applied  nature  of  the  IDT 
field.  It  is  likely  that  evaluation,  qualitative,  and  survey  data  collection  techniques  are  covered  in  multiple 
courses  when  students  are  learning  about  needs  assessment,  working  with  subject  matter  experts,  or  conducting 
formative  and  summative  evaluation  of  intervention. 

It  is  also  possible  that  experimental  and  quasi-experimental  methods  may  be  receiving  less  attention  in 
the  IDT  field  than  in  previous  years.  This  explanation  is  supported  by  a recent  analysis  of  ETR&D  (and  its 
forerunners)  that  indicated  the  use  of  descriptive  research  methods  has  increased  while  the  use  of  experimental 
methods  has  decreased  in  the  last  decade  (Ross  & Morrison,  2003).  Other  studies  have  suggested  that  consulting 
editors  of  ETR&D  have  a preference  for  applied  research,  case  studies,  and  developmental  methods  over  basic 
research  studies  (Klein,  1997).  Furthermore,  the  use  of  developmental  research  methods  has  increased  in  recent 
years  (Richey,  Klein,  & Nelson,  2003). 

Turning  to  research  processes,  we  found  most  IDT  students  are  taught  the  typical  steps  for  how  to 
conduct  a research  study  and  are  required  to  do  them  before  carrying  out  their  thesis  or  dissertation.  However, 
we  did  not  find  that  any  one  particular  textbook  as  being  most  widely  used  to  teach  students  about  these 
research  processes.  An  interesting  find  to  emerge  was  that  students  who  responded  to  our  survey  indicated  a 
lower  agreement  than  faculty  for  four  of  the  ten  research  processes  - analyzing  & interpreting  research  data, 
writing  research  reports,  sampling  participants,  and  developing  data  collection  instruments.  These  differences 
suggests  that  IDT  faculty  who  teach  research  courses  should  examine  their  objectives  and  activities  to  be  sure 
these  steps  in  the  research  process  areas  are  being  covered  in  enough  detail. 
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During  Phase  1 of  the  study,  we  were  surprised  to  find  that  only  few  courses  require  students  to  read, 
interpret,  and  analyze  primary  source  documents  such  as  published  research  articles.  However,  the  results  from 
Phase  2 suggested  that  most  IDT  students  are  expected  to  acquire  skills  related  analyzing  research  studies.  It 
may  be  that  our  survey  item  was  unclear  and  that  respondents  interpreted  it  to  mean  they  are  taught  to  analyze 
their  own  study  (or  their  students’  studies  in  the  case  of  faculty  respondents).  Regardless,  our  experience 
suggests  that  requiring  students  to  read,  interpret,  and  analyze  published  research  articles  is  a robust 
instructional  outcome  and  activity. 

We  also  think  that  students  should  learn  about  the  professional  contexts  of  research  in  the  IDT  field. 
However,  our  results  suggest  that  this  issue  may  not  be  covered  in  much  depth.  IDT  is  an  empirical  field; 
students  who  graduate  from  our  programs  should  be  able  to  apply  research  skills  to  a variety  of  contexts 
including  business,  industry,  military,  and  school-based  settings. 

Too  often,  research  is  thought  of  as  the  responsibility  of  academics  that  are  required  to  do  it  to  get 
tenure  and  promoted.  Faculty  who  offer  degrees  in  IDT  should  work  to  ensure  that  their  graduates  have 
competencies  to  be  successful  researchers.  They  should  also  push  students  to  apply  these  competencies 
regardless  of  the  setting  in  which  they  choose  to  work. 
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Hairy  Questions: 

The  Difference  a Bearded  Instructional  Character  Makes 
to  Middle  School  Students 

Theodore  Kopcha 
Leslie  Christine  Barrett 
Arizona  State  University 

Introduction 

There’s  no  keeping  up  with  what  teenagers  think  is  in  or  out;  status  changes  by  the  minute.  But  when 
you  develop  computer-based  instruction  (CBI)  for  that  population  of  users,  you’d  like  to  know  with  some 
reliability  that  the  program  is  in.  At  the  least,  you’d  like  to  know  that  users  won’t  respond  by  rolling  their  eyes. 

Such  was  the  case  with  a CBI  program  designed  for  a nonprofit  organization  that  demonstrates  the 
firing  of  a civil  war  cannon  for  middle  school  and  high  school  students.  The  organization  wanted  to  distribute  a 
CBI  program  to  schools  so  students  would  have  background  knowledge  about  the  cannon  before  watching  the 
firing  demonstration.  The  program  designer  decided  on  a character- mediated  approach  to  instruction,  with  the 
character  being  modeled  after  the  man  who  performs  the  demonstration.  That  man  is  a jovial  older  man  who 
wears  a beard  and  a replica  of  a Union  uniform;  the  CBI  program  character,  called  Sarge,  is  a line  drawing  in 
his  likeness. 

Preliminary  program  reviewers — adults — hypothesized  that  Sarge’s  beard  might  negatively  affect 
students’  responses  to  the  character.  Tobin  (2000)  conducted  a phenomenological  study  of  children’s 
generalizations  about  good  and  bad  characters  in  the  movie  The  Swiss  Family  Robinson.  He  found  that  children 
identified  the  movie’s  bad  guys,  in  part,  by  their  inferior  personal  grooming.  However,  within  a Civil  War 
context,  the  stereotypical  good  guy  certainly  might  have  grown  a little  scruff  on  his  face  after  months  on  the 
march. 

Stereotypes  might  be  crucial  to  interpreting  a character’ s purpose  in  a CBI  program  (Laurel,  1997). 
However,  internalizing  a culture’s  shared  expectations  for  archetypical  characters  is  a developmental  process 
(Applebee,  1978;  McKown  & Weinstein,  2003).  What  might  represent  grizzly  authenticity  to  an  adult  CBI 
designer  might  represent  wizened  obsolescence  to  a teenager.  To  investigate  what  a beard  might  mean  to 
students  both  within  a Civil  War  context  and  apart  from  that  context,  the  researchers  surveyed  middle  school 
students  about  their  responses  to  both  a bearded  Sarge  and  a clean-shaven  Sarge. 

Research  questions  are  as  follows: 

• Would  students  rather  have  a bearded  or  beardless  character  teach  them  about  the  Civil  War? 
Are  there  any  grade  level  or  gender  differences? 

• Do  students  perceive  a bearded  or  beardless  character  as  more  friendly?  Are  there  any  grade 
level  or  gender  differences? 

Research  by  Reeves  indicates  that  friendliness  is  an  essential  dimension  against  which  most  people  measure 
characters  (Reeves  & Greenberg,  1977;  Reeves  & Nass,  1996). 

Method 


Subjects 

Subjects  were  644  fifth-  through  eighth-grade  students  in  a suburban  middle  school  in  an  upper- middle 
class  community  in  a northeast  state. 

Materials  and  Measures 

This  study  used  a two-item  survey  posing  these  questions:  “Who  would  you  prefer  teach  you  about  the 
Civil  War?’’  and  “Who  do  you  think  is  the  friendliest?”  For  each  question,  students  chose  between  two  drawings 
of  Sarge  that  differed  only  by  the  presence  of  a beard  (see  Figure  1).  In  addition,  the  survey  collected 
information  about  the  students’  grade  levels  and  genders.  The  surveys  included  a written  introduction 
explaining  that  a college  student  was  designing  an  instructional  program  for  the  computer  and  that  the  college 
student  needed  their  help  in  deciding  on  a main  character  for  the  instruction. 

Four  forms  of  the  survey  counterbalanced  question  order  and  the  order  in  which  the  two  drawings 
appeared. 


487 


Procedures 

For  grades  six  and  seven,  a school  administrator  distributed  an  envelope  containing  25  survey 
instruments  to  teachers  during  a teacher  meeting  at  the  beginning  of  the  school  day.  Survey  instruments  were 
arranged  systematically  so  that  the  four  counterbalanced  survey  forms  would  be  dispersed  evenly  among 
students  in  each  classroom  Surveys  were  distributed  to  students  during  the  first  class  of  the  school  day  and 
returned  to  the  school  administrator  after  the  first  class.  Prior  to  administering  the  survey,  the  teacher  read  aloud 
to  the  class  the  survey’s  written  introduction.  Students  were  given  ten  minutes  to  complete  the  survey. 

For  grades  five  and  eight,  the  same  school  administrator  distributed  the  surveys  to  each  class  and 
followed  the  same  procedures  as  the  sixth-  and  seventh-grade  teachers.  The  administrator  collected  the  surveys 
immediately  after  completion. 


Figure  1.  Beardless  and  bearded  drawings  of  the  computer  character  called  Sarge. 


Design  and  Data  Analysis 

This  study  used  a two-way  design,  with  gender  and  grade  level  serving  as  status  variables.  Character 
choices  (i.e.,  bearded  Sarge  and  beardless  Sarge)  were  coded  as  zeros  and  ones,  with  a zero  representing  a 
preference  for  the  character  without  a beard  and  a one  representing  a preference  for  the  character  with  a beard. 
Therefore,  mean  scores  for  the  dependent  variable  could  be  interpreted  as  the  percentage  of  students  selecting 
the  bearded  character.  Binomial  tests  were  run  to  determine  if  the  preference  proportions  differed  significantly 
from  the  chance  level  of  .50.  Binomial  tests  were  conducted  for  the  entire  sample  and  for  each  level  of  the 
gender  and  grade  level  variables  within  the  separate  question  contexts. 

To  assess  the  effects  of  grade  level  and  gender  on  students’  preferences  for  a bearded  character,  the 
researchers  used  2 (gender)  x 4 (grade  level)  analyses  of  variance  (ANOVAs)  with  gender  and  grade  level  as 
between-subjects  variables.  Analysis  of  dichotomous  data  in  this  manner  has  been  shown  to  be  justified  by  the 
robustness  of  ANOVA  (Glass,  personal  communication,  February  24,  2004;  Glass,  Peckham,  & Sanders,  1972). 
Those  ANOVAs  were  run  for  each  of  the  two  survey  questions.  The  inclusion  of  question  context  as  an 
independent  variable  in  those  ANOVAs  would  have  complicated  the  interpretation  of  any  gender  or  grade  level 
effects.  Therefore,  the  difference  between  survey  questions  (or  question  contexts)  was  analyzed  separately,  with 
a paired-samples  /-test.  Because  of  the  number  of  analyses,  a was  set  at  .01  for  all  statistical  tests. 

Results 

Table  1 shows  the  mean  proportions  of  all  students  choosing  the  bearded  character  overall,  within  each 
character  context , and  by  grade  level  and  gender.  Table  l’s  totals  column  shows  that  for  both  questions 
combined  students  chose  the  bearded  character  more  frequently  than  the  bearded  character.  This  overall 
preference — 54  percent  of  choices  for  the  bearded  character — was  significantly  different  from  chance,  P = .54, 
p < .01.  In  addition,  a significant  proportion  of  students  chose  the  bearded  character  when  asked,  “Who  would 
you  prefer  teach  you  about  the  Civil  War?”,  P = .74,  p < .01.  Only  34  percent  of  students  chose  the  bearded 
character  when  asked,  “Who  do  you  think  is  the  friendliest?”  Student’s  choice  of  the  beardless  character  as 
more  friendly  also  was  significantly  different  from  chance,  P = .34,  p < .01. 
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Table  1 Students’  choices  for  a bearded  character  as  a Civil  War  teacher  and  as  being  friendlier  than  a 
beardless  character:  Mean  proportions  by  grade  level  and  gender 


Gender 

Grade  level 

Totals 

Male 

Female 

5 

6 

7 

8 

Civil  War  teacher 

.80 

.67 

.68 

.74 

.80 

.75 

.74 

Friendliness 

.38 

.30 

.31 

.29 

.35 

.41 

.34 

Totals 

.59 

.48 

.49 

.52 

.57 

.58 

.54 

When  asked  to  select  which  character  was  more  friendly,  both  males  (62  percent)  and  females  (70 
percent)  chose  the  beardless  character  more  frequently  than  the  bearded  character.  All  grade  levels  also  more 
frequently  chose  the  beardless  character  as  being  more  friendly.  Choice  proportions  for  the  beardless  character 
ranged  from  70  percent  of  sixth-graders  to  59  percent  of  eighth-graders. 

Within  the  context  of  selecting  a Civil  War  teacher,  both  males  (80  percent)  and  females  (67  percent) 
chose  the  bearded  character  more  frequently  than  the  beardless  character.  All  grade  levels  also  preferred  the 
bearded  character  more  often  than  the  beardless  character  as  a Civil  War  teacher;  proportions  ranged  from  68 
percent  of  fifth-graders  to79  percent  of  seventh-graders.  All  grade-level  and  gender  groups’  mean  preference 
proportions  for  both  questions  were  significantly  different  from  chance. 

No  significant  differences  were  obtained  for  the  grade  level  main  effect.  However,  significant 
differences  were  found  within  gender  on  both  questions.  On  the  Civil  War  teacher  question,  males  more 
frequently  preferred  the  bearded  character,  F(  1,  636)  = 13.1 1,  p < .01.  Males  also  more  frequently  chose  the 
bearded  character  as  being  friendly,  F(l,  635)  = 4.12,p  < .05.  No  significant  interactions  were  found. 

Based  on  a paired-samples  r-test,  question  context  was  found  to  have  a significant  effect  on  students’ 
choices  of  the  bearded  character,  t = -15.76,  p < .01.  Students  more  frequently  chose  the  bearded  character  as  a 
Civil  War  teacher  than  they  chose  that  character  as  being  the  more  friendly  character. 

Discussion 

Interpreted  simply,  students  prefer  a bearded  character  over  a beardless  character  when  considering 
who  they  want  to  teach  them  about  the  Civil  War  (74  percent  choosing  the  bearded  character).  However,  when 
considering  who  is  more  friendly,  students  more  frequently  choose  the  beardless  character  (66  percent  choosing 
the  beardless  character).  Each  of  those  preferences  was  fairly  strong.  An  examination  of  the  effect  of  context  on 
children’s  choices  of  characters  is  helpful  for  interpreting  what  those  results  mean  for  instructional  design. 

Context  Differences 

The  proportions  of  students  choosing  the  bearded  character  were  significantly  different  for  the  two 
different  survey  questions.  “Who  do  you  think  is  the  friendliest?”  asked  for  a more  general  indication  of 
preference,  an  indication  of  which  character  was  perceived  as  more  affable,  more  likeable,  more  preferable. 
Most  students  chose  the  beardless  character  as  being  more  friendly. 

The  question  also  aimed  to  explore  the  personality  traits  that  students  might  associate  with  beards.  The 
researchers  hypothesized  that  there  might  be  two  bearded-men  stereotypes  on  opposite  ends  of  the  friendliness 
spectrum.  First,  there’s  the  Santa  archetype;  jolly,  generous,  and  friendly.  Then  there’s  the  mountain  man 
stereotype;  gruff,  grizzly,  and  much  less  friendly.  A beard  likely  carries  connotations,  the  researchers  just 
weren’t  sure  what  those  connotations  would  be  for  this  study’s  adolescent  students,  they  weren’t  sure  what 
stereotypes  those  participants  held  about  bearded  men.  In  contrast,  the  beardless  character  was  relatively 
stereotype-free,  meant  to  represent  an  “average  guy.” 

The  results  do  not  permit  a definitive  statement  that  students  perceived  the  bearded  character  as 
unfriendly,  but  the  results  to  permit  it  to  be  said  that  friendliness  was  a personality  trait  that  students  associated 
the  trait  of  friendliness  less  strongly  with  the  bearded  character  than  with  the  beardless  character.  Because  facial 
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hair  was  the  only  attribute  that  varied  between  the  two  characters,  the  results  seem  to  indicate  that  beards,  in 
isolation  from  all  other  attributes  of  animated  characters,  do  not  strongly  transmit  an  air  of  friendliness.  That 
finding  appears  to  complement  Tobin’s  (2000)  findings  that  children  named  less-than -perfect  grooming  as  a 
sign  that  a character  is  a bad  guy. 

Where  the  friendliness  question  was  context -neutral,  the  question  “Who  would  you  prefer  teach  you 
about  the  Civil  War?”  associated  the  character  with  a rich  context,  and  the  question  elicited  very  different 
results:  given  that  scenario,  most  students  preferred  the  bearded  character.  Many  approaches  to  design  suggest 
placing  students  in  authentic  or  simulated  contexts  (Choi  & Hannafin,  1995;  Cognition  and  Technology  Group, 
1992).  In  addition,  some  researchers  and  developers  of  animated  agents  suggest  that  agent  characters  look  their 
parts,  that  they  represent  their  role  in  the  software  (Laurel,  1997).  A bearded  character  might  have  appeared 
more  authentically  Civil  War  era;  students  might  have  seen  pictures  of  Civil  War  figures  or  movies  about  the 
Civil  War  and  noted  that  many  of  the  men  in  that  time  wore  beards.  A bearded  character  also  might  have  better 
represented  the  Civil  War  context;  beards  typically  are  associated  with  older  men  and  the  Civil  War  is  a 
historical  (i.e.,  old)  context. 

If  in  fact  students  chose  the  bearded  character  because  of  his  authenticity  or  because  of  what  he  might 
have  represented,  the  results  provide  evidence  that  aspects  of  authenticity  and  representation  are  important  to 
students.  Given  the  context  of  learning  about  the  Civil  War,  having  an  authentic -looking  animated  instructor 
(i.e.,  a bearded  Sarge)  appeared  to  be  more  important  to  this  study’s  participants  than  having  a friendly -looking 
animated  instructor  (i.e.,  a beardless  Sarge). 

Gender 

For  both  questions,  boys  more  frequently  preferred  the  bearded  character  than  girls.  Previous  studies 
have  shown  students  to  prefer  characters  of  the  same  gender  that  they  are  (Barrett  & Sullivan,  2004;  Beyard- 
Tyler  & Sullivan,  1980).  In  this  study,  female  students  did  not  have  the  choice  of  a female  character.  However, 
it  might  be  argued  that  a beard  makes  a male  character  even  more  masculine.  If  that  is  the  case,  then  female 
students  more  frequently  preferred  the  less  masculine  character  than  male  students , and  male  students  more 
frequently  preferred  the  more  masculine  character.  That  hypothesis  suggests  that  prior  studies  of  preference  for 
character  gender  were  too  simplistic;  preference  might  need  to  be  studied  for  characters  that  fall  on  various 
points  of  a continuum  of  masculinity  and  femininity. 

An  alternate  explanation  might  simply  be  that  a beard  represents  a certain  ruggedness,  toughness,  or 
masculine  maturity.  Those  traits  certainly  would  have  been  more  appealing  to  male  participants  than  female 
participants. 

Implications  for  Design 

Studies  of  gender  preferences  (Barrett  & Sullivan,  2004;  Beyard-Tyler  & Sullivan,  1980)  make  this 
rule  clear:  When  designing  instruction  for  females  use  female  characters;  when  designing  instruction  for  males 
use  male  characters.  However,  instruction  for  users  of  only  one  gender  is  rare.  This  study  suggests  that 
characters  might  be  perceived  as  falling  on  a gender  continuum.  Therefore,  to  appeal  to  users  of  both  genders 
using  characters  who  are  extraordinarily  girly  girls  or  manly  men  likely  should  be  avoided. 

That  context  and  character-authenticity  appear  to  be  important  to  students  makes  the  designer’s  job 
both  easier  and  tougher.  When  designing  educational  software  in  which  an  animated  character  will  be  used,  the 
designer  simply  might  need  to  consider  the  content  and  learning  environment  then  create  a character  that  fits 
both.  Nevertheless,  designers  need  to  be  aware  that  children  might  not  share  the  same  set  of  concepts  and 
character  archetypes  that  most  adults  of  a culture  might  share  (Applebee,  1978;  McKown  & Weinstein,  2003). 
In  addition,  the  designer  must  be  wary  of  reusing  characters;  contexts  will  vary  from  instructional  program  to 
instructional  program.  However,  because  developing  animated  characters  for  software  is  an  expensive 
enterprise,  designers  might  need  to  reuse  characters.  When  this  is  the  case,  designing  a character  that  fits  only 
certain  contexts  should  be  avoided. 

As  suggested  by  this  study,  authenticity  and  desirability  might  not  always  work  together;  an  authentic 
Civil  War  character  might  not  be  perceived  as  the  friendliest  of  characters.  If  the  designer  is  especially 
dedicated  to  meeting  the  needs  of  learners,  experimentation  must  be  done  to  find  the  right  combination  of 
character  attributes  to  convey  both  authenticity  and  desirability. 
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Abstract 

Classic  research  methods  are  ineffective  in  capturing  the  dynamic  relationships  among  users, 
technology,  and  outcomes  for  technology  integration  research.  Cognitive  research  attempts  to 
eliminate  environmental  variability  and  test  for  recall  and  transferability  of  knowledge 
representations.  Such  approaches  lead  to  arguably  ineffective  measures  of  learning.  Activity  theory 
provides  a holistic  framework  to  investigate  relationships  among  the  elements  present  in  a technology 
integration  activity.  These  research  approaches  are  compared.  A CHAT  framework  is  suggested  as  an 
appropriate  structure  for  analyzing  technology  integration  efforts.  This  approach  focuses  on  the 
dynamic  relationships  among  individuals,  goals,  tools,  community  members,  and  mediating  factors 
that  are  the  elements  of  any  human  activity.  An  example  of  this  framework  in  practice  is  presented.  A 
research  model  generated  from  applying  CHAT  approach  is  proposed. 

Introduction 

As  educational  technology  has  become  more  prevalent  in  educational  settings,  research  efforts  have 
increased  to  study  technology’s  uses  and  impacts.  Although  by  nature  technology  is  a tool  used  within  human 
activity,  the  traditional  research  approaches  tend  to  focus  only  on  the  user  of  the  technology,  specifically 
outcomes  as  a result  of  using  technology  (Matheson  et  al.,  1999).  In  an  attempt  to  eliminate  the  variability  of  the 
environment  investigators  ignore  or  control  for  key  elements  of  the  activity  itself  such  as  historical  background 
and  motivations  of  the  subjects,  the  technology’s  role  in  achieving  goals,  and  relationships  among  others  within 
the  activity.  Activity  theory  is  a socio -cultural  and  historical  lens  through  which  human  activity  systems  can  be 
holistically  analyzed  (Engestrom,  1999;  Jonassen  & Rohrer-Murphy,  1999).  This  approach  focuses  on  the 
interaction  of  human  activity  and  human  thought  within  its  relevant  environmental  context.  Since  learning  is  not 
a precursor  to,  rather  it  emerges  from  activity,  research  should  attempt  to  examine  the  individuals(s)  involved  in 
the  activity  and  activity  elements  such  as  the  product  of  the  activity,  mediating  tools,  community  members,  and 
guiding  rules  while  the  individual(s)  is/are  acting  on  and  attempting  to  produce  an  outcome.  This  paper  provides 
a comparison  of  CHAT  and  traditional  cognitive  research  methods  and  an  example  of  how  CHAT  was  used  to 
analyze  the  impact  of  a technology  integration  intervention.  A research  model  generated  from  this  project  is 
also  proposed  and  discussed. 

Brief  Overview  of  Activity  Theory 

Activity  theory  adds  value  to  assessment  processes  in  that  it  suggests  that  the  combined  foundational 
elements  of  an  activity  are  the  unit  of  analysis  that  represents  the  minimum  elements  of  an  object-oriented, 
collective,  and  culturally  mediated  human  activity  (Engestrom,  1987).  It  is  the  internal  tensions  and 
contradictions  within  and  among  the  elements  of  a human  activity  that  lead  to  the  transitions  and  transformation 
of  knowledge.  The  basic  elements  of  an  activity  include  subject,  object,  tools,  community,  rules,  and  division  of 
labor  (Engestrom,  1987;  Kuttii,  1991;  Kuttii,  1996). 

The  main  focus  of  any  activity  is  in  the  production  of  an  outcome  (object),  physical  or  mental.  The 
subject,  an  individual  or  group,  determines  that  there  is  a need  or  motive  to  fulfill,  the  object.  Using  the  tools 
(e.g.,  technology,  training,  conceptual  ideas,  people)  the  subject  moves  toward  accomplishing  the  object. 

The  community  members  set  rules  and  norms  under  which  the  subject  operates  and  establishes  how  the 
community  members  organize  (division  of  labor)  to  meet  goals.  All  of  the  elements  influence  the  others  and  are 
influenced  by  social,  cultural,  and  historical  factors,  such  as  background  knowledge,  personal  bias,  availability 
of  tools,  and  other  factors.  Each  individual  activity  is  also  affected  by  other  surrounding  activities  that  may  have 
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a primarily  tool,  community,  rule,  or  some  other  activity  element  focus.  Thus,  activity  has  motive  and  is 
complex,  dynamic,  historically -driven,  and  transforming.  See  figure  1. 


Tools 


Figure  1.  Activity  Theory  framework 

Comparison  of  Traditional  Cognitive  Research  and  CHAT  Paradigms 

Traditional  cognitive  research  paradigm.  It  is  argued  that  most  educational  research  focusing  on 
cognitive  development  (learning)  takes  a narrow  view  that  does  not  represent  the  true  complexity  of  the  learning 
process  (Gay  & Bennington,  1999).  Learning  is  defined  as  a change  in  schema  precipitated  by  sensory  reception 
and  active  manipulation  of  new  information  until  new  information  is  memorized,  stored,  and  readied  for  recall 
in  existing  mental  structures  (schema).  During  this  type  of  research  there  is  a struggle  to  reduce  the 
environmental  variables  in  the  study  environment  that  influence  the  hypothesized  cognitive  changes  so  that 
findings  can  be  generalized  to  specific  treatments  or  environmental  variables.  Such  research  has  included 
investigating  outcomes  measures  like  attitudes  toward  use  of  technology  in  the  classroom  (Koszalka,  2000; 
Koszalka,  2001),  teachers’  views  and  beliefs  of  technology  related  to  teaching  practices  (Dexter,  Anderson,  & 
Becker,  1999;  Honey  & Moeller,  1990),  and  teacher  (e.g.,  skill  level)  and  classroom  factors  (e.g.,  access)  as 
predictors  of  technology  use  (Becker,  1999).  This  approach  generally  attempts  to  control  the  multiple  factors 
that  may  have  influenced,  and  are  currently  influencing,  change  and  structure  of  knowledge  (Mathison  et  al., 
1999).  See  table  1. 

Cultural  historical  activity  theory  research  paradigm.  In  an  activity  theory  approach  each  activity  is 
analyzed  as  part  of  the  collective  and  with  a social-historical  context  of  the  individual  and  the  collective,  thus 
CHAT.  This  approach  requires,  at  minimum,  a shared  understanding  of  the  character  and  history  of  the  subject, 
the  object  unto  which  the  individual  is  attempting  to  reach,  the  characteristics  of  the  surrounding  community, 
and  the  tools  available  to  the  subject.  The  focus  of  the  analysis  is  on  the  interaction  of  human  activity  and  the 
whole  of  the  individual’s  mentality  as  they  interact  within  a relevant  environmental  context.  Activity  theory  is 
thus  a framework  for  understanding  the  totality  of  human  activity  in  context  (Bodker,  1991). 

Traditional  cognitive  and  CHAT  paradigms  hold  different  points  of  views  toward  research  related  to  human 
learning.  Each  suggests  differences  in  the  how  they  define  (i)  learning  (ii)  technology’s  role  (iii)  assessment 
focus  (iv)  assessment  context  (v)  evaluator’s  role  and  (vi)  data  collection  context.  Those  who  use  traditional 
cognitive  paradigms  believe  that  learning  is  a permanent  change  in  schema  that  occurs  through  assimilating  and 
accommodating  external  information  into  schema.  During  the  learning  process,  technology  provides  the 
information  that  the  learner  acquires.  Thus,  this  paradigm  assumes  that  improving  technology  use  can  result  in 
the  facilitation  of  learning.  Assessment  research  focuses  on  the  changes  in  the  subject  as  a result  of  using 
technology.  Refer  to  Table  1. 

The  CHAT  paradigm  argues  that  learning  is  a process  of  constant  interaction  with  the  environment  and 
others.  Knowledge  is  constructed  by  individual  learners,  building  on  existing  historical  experiences,  within  the 
learners’  context . Technology  is  a mechanism  to  actively  engage  learners  in  the  learning  process,  the  use  of 
technology  is  influenced  by  the  rules  of  and  interactions  with  the  community,  and  it  is  a tool  that  mediates 
learning  activities  with  which  to  construct  individual  knowledge.  Thus,  the  CHAT  paradigm  assumes  that 
outcomes  (knowledge)  are  constructed  by  interaction  within  an  activity  among  useis,  technology,  and 
environmental  factors  all  within  a context . CHAT  assessment  research  therefore  focuses  on  understanding  the 
interaction  process  of  the  activity  within  the  naturalist  environment.  Thus  this  research  provides  a more  holistic 
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description  of  the  knowledge  construction  activities. 

Table  1 . Comparison * of  traditional  cognitive  paradigm  and  CHAT  paradigm 


Traditional  Cognitive  Paradigm 

CHAT  paradigm 

Definition  of 
learning 

Learning  is  permanent  change  in  schema 
(assimilation  and  accommodation  to 
existing  memory  structures).  Knowledge 
is  transferable  from  one  individual  to 
another. 

Learning  is  a process  of  constant 
interaction  with  the  environment  and 
others.  Knowledge  is  constructed  by 
individual  learners,  built  on  historical 
experiences,  within  his  or  her  context , 
knowledge  is  not  transferred,  rather  it  is 
constructed  differently  in  all  individuals 

Technology’s 
role  in  learning 

The  technology  provides  information  by 
simulating  the  initial  sensory  perception 
in  users’  cognitive  process  and  provides 
mechanisms  with  which  the  user  can 
manipulate,  organize,  and  represent 
information  in  ways  that  will  prompt 
memory  storage 

Technology  provides  user  with 
mechanisms  to  actively  engage  in  the 
learning  process,  access  multiple  forms 
and  perspectives  of  information,  think 
critically,  communicate  during  inquiry, 
and  engage  in  other  activities  with  which 
to  construct  own  knowledge. 

Assessment 

focus 

Technology  itself  — the  evaluators  are 
dedicated  to  improving  the  technology 
because  they  believed  improvements  in 
the  technology  will  result  in  the  high 
quality  learning  outcomes. 

The  interaction  between  users  and 
technology  — the  evaluators  are  dedicated 
to  understanding  and  creating  better 
interaction  between  technology  and  users 
because  they  believe  that  outcomes 
(knowledge)  are  constructed  by  an 
interaction  among  user  and  technology. 

Assessment 

context 

Assessment  is  conducted  within  the 
control  environment,  excluding  any 
factor  other  than  technology  itself,  reduce 
environmental  variability 

Assessment  is  conducted  within  the 
natural  environment  (same  context  as  if 
the  user  were  using  the  technology  ‘at 
work,’) considering  users’  social-cultural, 
historical,  and  technology  background. 

Evaluators  ’ role 

Outsiders  to  objectively  judge  the 
evaluation  results. 

Facilitators  in  the  assessment  process  and 
interactions  among  user  and  technology. 

Data  collection 
context 

Controlled  environments  and 
interventions.  Data  collection  techniques 
are  used  to  gather  information  based  on 
self-reports  and  participants’  recall.  Talk 
-alouds  are  used  to  gather  data  on  the 
users  thoughts  during  technology  use  (no 
interaction  with  evaluator) 

Data  collection  occurs  within  the 
naturalist  environment,  using  technology 
as  a collection  tool.  Web  logs,  interactive 
talk-alouds,  observations  of  technology  in 
practice  techniques  are  used  to  collect 
data  on  the  natural  interaction  between 
technology  and  user. 

* Adapted  from:  Matheson  et  al.,  1999 


A Case  Study  of  CHAT  to  Investigate  Technology  Integration 

Research  Context 

The  Initiative  to  Develop  Education  through  Astronomy  and  Space  Science  (IDEAS)  project  was 
initiated  to  promote  enrichment  of  science,  mathematics,  and  technology  education  through  the  use  of  NASA's 
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mission-based,  technology -based  astronomy  and  space  science  research  resources.  The  object  was  to  help  K-12 
educators  (subjects)  develop  computer  technology  integration  skills  and  integrate  NASA  space  science  and 
astronomy  web  resources  into  their  classrooms  to  enhance  teaching  and  learning.  As  such  a teacher  academy 
was  developed  to  (1)  immerse  educators  in  astronomy  topics,  (2)  expose  teachers  to  NASA  space  science  and 
astronomy  web  resources,  (3)  train  teachers  in  computer  technology  integration  techniques,  (4)  help  teachers 
develop  strategies  to  reduce  barriers  for  computer  use,  and  (5)  provide  teachers  with  time  to  create  lessons  that 
integrated  NASA  web  resources  into  their  classrooms.  The  4-day  academy  was  followed  by  2-years  of 
observation  and  follow-up  support  for  the  first  cohort  and  1 year  of  follow-up  for  the  second  cohort,  from  which 
the  data  were  collected  on  classroom  environment,  teaching  strategies,  NASA  web  resources  use,  and 
technology  integration  practices. 

CHAT  was  used  to  both  inform  and  create  a research  framework  to  facilitate  collections  and  analysis 
of  the  data.  See  figure  2.  For  example,  the  CHAT  framework  helped  to  identify  the  factors  that  were  measured 
including  previous  training  and  experiences  of  the  subjects’  before  participation,  interactions  with  peers  and 
administrators  during  technology  implementation,  established  policies  and  support  structures  for  teaching  and 
technology,  and  changes  in  availability  of  technology  and  curriculum  resources  throughout  the  data  collection 
period,  to  name  a few. 
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Figure  2.  CHAT  research  and  data  collection  framework 
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Methods 


Research  Questions 

The  following  research  questions  were  investigated  using  a case  study  approach: 

1.  How  did  teaching  methods,  technology  integration  strategies,  and  use  of  NASA  astronomy  and  space 
science  resources  change  over  time? 

2.  What  elements  of  the  academy  affected  educators’  teaching  methods,  technology  integration  strategies, 
and  use  of  NASA  resources  overtime? 


Subjects 

The  participating  educators  were  from  schools  within  an  urban  school  district  in  upstate  New  York. 
School  administrators  were  asked  to  help  recruit  middle  and  high  school  science  and  math  teachers,  of  which 
the  subjects  volunteered  to  participate.  In  the  first  year  7 educators  participated.  In  year  2 there  were  1 1 new 
subjects  from  the  district.  The  teachers  were  required  to  have  at  least  3 years  science  or  math  teaching 
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experience,  basic  computer  and  internet  skill,  and  access  to  an  internet-connected  computer  at  school. 


Data  Sources  and  Analysis 

Five  sources  of  data  were  collected  on  each  of  the  educators  during  the  academy  and  in  follow-up 
observations  in  the  classroom:  (1)  initial  background  survey,  (2)  workshop  evaluations,  observations,  and 
feedback,  (3)  classroom  observations  and  photographs,  (4)  mid-year  survey  and  (5)  on-going  interviews.  The 
surveys  probed  for  attitudes,  perceptions,  practices,  and  demographic  information  of  the  teachers.  The 
observations  focused  on  classroom  environmental  factors  and  teaching  practices  including  technology  and 
resources  use,  teaching  methods,  and  classroom  events.  The  interviews  were  used  to  solicit  feedback  on 
teaching  technology,  and  resources  uses  and  issues,  as  well  as  teachers’  thoughts,  ideas,  and  explanations  of 
their  practice.  The  researchers  collected  data  both  in  the  etic  (as  the  outsider)  and  emic  (engaging  with  the 
teachers  and  students  in  the  classroom).  These  data  points  were  used  to  identify  the  trends  of  changes  in 
teaching  and  technology  and  NASA  resource  integration. 

Quantitative  data  were  organized  to  describe  of  the  entire  group  of  educators.  The  complete  set  of  data 
was  analyzed  to  identify  interactions  among  the  teachers,  tools,  community,  and  objectives  identifying  trends  on 
both  individual  and  group  levels.  Data  regarding  the  educators’  teaching  methods,  use  of  technology,  and  use  of 
NASA  web  resources  were  tracked  using  a time  series  approach  to  capture  temporal  changes.  A profile  for  each 
teacher  was  created  and  used  to  identify  resulting  themes. 

Results 

The  two  cases  presented  here  were  chosen  from  a representative  sample  of  educators  who  participated 
in  the  initial  year  of  the  project.  These  two  cases  were  selected  based  on  maximal  variety  of  participants  ’ 
responses  to  the  initial  survey.  The  two  cases  described  in  this  paper  were  selected  to  represent  the  extremes  of 
teaching  experience  and  web  use.  See  Table  2. 

Table  2.  Baseline  demographics  for  first  year  participating  educators. 


Educator 

Initial 

Attitude  score 
(-30  to  +30) 

Teaching 
exp.(<5  yrs  / +5 
yrs) 

Self-rated  computer 
expertise 

(low  to  experienced) 

Self-reported 
web  use 

(lesson  prep,  in  class) 

1* 

-8 

+5 

Experienced 

Weekly  for  prep/class 

2* 

10 

<5 

Low 

Rarely  for  prep  only 

3 

19 

+5 

Intermediate 

Monthly  for  prep/class 

4 

12 

+5 

Intermediate 

No  data 

5 

10 

+5 

Low 

Monthly  for  prep/class 

6 

24 

+5 

Intermediate 

Weekly  for  prep/class 

7 

23 

<5 

intermediate 

Monthly  for  prep  only 

* educators  included  in  comparison. 
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Educator  1 profile 

Educator  1 participated  in  the  project  for  two  years.  She  taught  6th  grade  science  for  more  than  5 years. 
She  spent  more  than  60  percent  of  time  teaching  science  and  less  than  20  percent  of  the  time  on  mathematics 
topics . Teacher  1 self -identified  her  primary  teaching  strategy  as  inquiry  and  indicated  that  she  also  uses  hand- 
on  activities,  collaborative  work,  and  problem-based  learning  regularly.  In  the  baseline  survey,  she  rated  herself 
as  an  experienced  web  user,  holding  a slightly  negative  attitude  toward  web  use  in  education  (-8  on  a scale  of  - 
30  to  +30).  Data  collected  during  initial  observations  indicated  that  her  classroom  environment  contained  one 
computer  with  web  access,  a printer,  instructional  TV,  and  an  overhead  projector.  She  indicated  that  she  often 
used  the  computer  and  web  to  search  for  school-related  information  and  to  prepare  lesson  weekly,  however 
rarely  used  technology  in  the  classroom  with  students.  See  figure  4. 
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Figure  4.  Trends  for  educator  1 

Educator  l’s  frequently  communicated  with  the  IDEAS  support  team  and  other  educators  from  the 
academy  to  share  teaching  information  from  the  project  and  her  new  ideas  for  technology  use.  In  addition,  at  her 
request,  the  IDEAS  support  team  provided  her  with  a portable  SMARTBoard  and  projector  and  technology 
support  to  operate  these  new  resources , additional  NASA  web  resources  for  specific  science  topics,  reflection 
tools  to  help  her  think  more  about  how  to  use  such  technologies , and  use  of  the  IDEAS  online  lesson  plan 
templates.  She  also  shared  examples  of  the  lessons  she  created  as  a result  of  year  1,  NASA  resources  and  web 
resources  she  found  most  useful  given  her  teaching  objectives,  and  examples  of  student  projects  that  were 
produced  from  her  classroom  with  the  year  2 participants. 

The  two-year  observations,  summarized  in  figure  4,  indicated  several  changes  throughout  project 
participation.  She  developed  higher  attitude  scores,  used  NASA  tools  and  resources  more  frequently,  and  made 
changes  in  her  teaching  methods  and  the  types  of  resources  she  used  in  her  classroom  over  the  two  years.  There 
was  an  increase  in  the  number  of  NASA  posters  and  other  resources  used  to  ‘decorate’  her  classroomand  that 
she  used  to  engage  her  students  in  activities  and  discussions.  In  a final  debrief  educator  1 reported  that  she  now 
begins  all  web  searches,  to  support  any  science  topic,  at  the  NASA  search  site.  Although  rarely  using  this 
approach  before  participating  in  the  IDEAS  project,  teacher  1 also  began  incorporating  web  resources  more 
regularly  into  her  teaching  and  engaging  her  students  with  the  web  to  support  their  learning. 

Educator  2 profile 

A similar  profile  was  created  for  educator  2,  showing  his  activities  within  the  academy  and  classroom 
and  how  his  teaching,  technology,  and  NASA  resources  practices  and  uses  changed  over  time.  These  data  also 
provide  evidence  of  the  factors  within  the  activity  may  have  influenced  noted  changes. 

Summary  of  Results 

Many  changes  were  observed  in  both  of  the  subjects  with  regard  to  teaching,  technology,  and  NASA 
resources  practices.  These  changes,  and  lack  of  changes,  were  traced  to  historical  and  activity  factors  from  the 
academy  and  classroom  environment.  Factors  included  development  of  new  knowledge  of  strategies  and 
resources  gained  during  the  academy,  project  and  school  support  mechanisms  and  resources,  curriculum 
requirements,  peer  collaborations  and  support,  sharing  of  ideas  within  and  outside  the  subject  cohort,  classroom 
contextual  factors  such  as  room  layout  and  technology  access,  school  policy  such  technology  rules  and 
regulations,  and  teachers’  personal  perceptions,  attitudes,  and  experiences. 

Discussion 

The  data  collected  was  rich  and  full  of  illustrative  stories  that  explained  emerging  patterns  and  how 
these  teachers  interacted  with  the  resources,  community,  and  objectives  for  which  they  were  striving.  Many 
examples  described  how  teachers  helped  each  other,  reorganized  their  classrooms,  accessed  new  technologies 
and  resources  they  had  not  previously  thought  about,  worked  through  issues  of  educational  technology  union 
rules,  identified  new  support  resources  at  their  schools,  and  generally  changed  the  way  they  were  thinking  about 
teaching  and  resources  use.  These  two  educators  for  example,  similar  to  the  other  participating  educators  from 
both  years  of  the  academy,  began  to  make  changes  in  their  teaching,  technology,  and  NASA  resources  use 
strategies.  The  academy  itself,  and  follow-up  support,  seemed  to  have  set  off  a variety  of  activities  and 
interactions  that  supported,  or  perhaps  influenced,  these  educators’  changes  over  time.  Four  main  themes 
emerged:  teaching  methods,  technology  integration,  NASA  resources  are  rich  sources  on  information,  and 
NASA  resources  as  technology-based  resources. 

Theme  1 - movement  toward  more  student-centered  methods.  The  most  commonly  used  (reported  and 
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observed  baseline)  teaching  methods  prior  to  the  academy  were  presentation  and  hands-on  methods.  Both 
inquiry  and  collaborative  learning  emerged  after  the  academy  in  both  of  the  studied  classrooms.  Both  teachers 
began  to  use  inquiry  to  prompt  students  to  associate  current  events,  daily  life,  and  their  experiences  with  new 
concepts.  In  addition,  the  educators  prompted  the  students  more  to  define  important  concepts  and  rephrase  their 
understanding  of  the  content  in  their  own  words.  These  inquiry  sessbns  were  often  accompanied  with 
discussions  on  examples  of  NASA  scientists  at  work,  for  example  why  it  was  important  for  NASA  mission 
scientists  to  understand  weather  topics,  communicate  during  major  projects,  and  use  similar  metrics.  Thus  the 
use  of  the  new  resources  and  technologies  seems  to  have  had  some  affect  on  the  methods  the  teachers  began  to 
use  to  further  engage  students  in  the  classroom. 

Theme  2 - movement  toward  technology  enhancements.  These  educators  did  not  have  the  same 
technology  available  for  use  in  their  classrooms,  yet  both  used  the  web  to  help  them  prepare  for  lessons.  The 
data  indicated  however,  that  both  also  began  to  use  technology  as  a media  to  present  content  and  gather  and 
print  teaching  resources.  The  web  was  used  more  often  at  the  end  of  the  project  by  both  educators  to  search  for 
additional  resources  that  would  be  used  to  support  their  teaching  and  as  a presentation,  prompting,  or 
exploration  tool  in  their  classroom. 

Theme  3 - inclusion  of  ‘richer’  information  resources.  Both  educators  began  to  use  NASA  resources  in 
three  ways:  teaching  resource  (supportive  of  presentation,  inquiry,  activity),  motivational  classroom  decorations 
(supportive  of  explanation  and  inquiry),  and  as  a sharing  tool  with  peers  (sharing  new  resources  and  lesson 
ideas).  They  incorporated  more  illustrative  (pictures)  resources  in  their  teaching  approaches  to  help  students 
visualize  content.  They  both  began  to  use  NASA  resources  and  websites  more  frequently,  including  lesson 
plans,  information  related  to  the  content,  and  vocabulary,  when  preparing  to  teach.  Both  educators  also  shared 
several  web  resources  with  peers  and  students  across  the  school  district. 

Theme  4 - engaging  with  technology  resources  for  teaching  and  learning.  Both  educators,  provided 
with  the  NASA  resources  and  technology  by  the  IDEAS  project,  were  motivated  and  increased  their  use  of 
NASA  resources  and  technology  into  their  teaching.  The  NASA  resources  enriched  both  teachers  ’ access  to 
scientifically  accurate  resources  and  their  teaching  methods.  Additionally,  both  teachers  made  use  of  more 
technology  to  aid  in  their  searching  and  preparation  activities  and  in  presenting  information  and  activities  to 
students . 

The  technology  integration  efforts  resulted  in  a complex  process  of  interactions  with  tools  and 
community  members.  The  use  of  computer  technology  and  NASA  resources  seemed  to  provide  motivational 
context  for  many  topics  from  studying  weather  and  measurement  to  creative  writing.  Many  of  the  new  ideas 
were  inspired  by  examples  presented  in  the  academy  or  from  collaborations  with  peers  and  the  IDEAS  support 
team.  Yet,  as  illustrated  by  these  two  different  educators  one  engaged  a great  deal  with  the  IDEAS  team  and  the 
other  only  slightly  to  allow  classroom  observations  and  brief  interviews.  Therefore,  the  influence  of  the  IDEAS 
academy  and  personnel  may  not  be  the  major  factor  in  the  changes  observed.  Introduction  to  the  content  and 
resources  of  technology  integration  models  and  NASA  resources  and  the  characteristics  of  the  teachers  and  their 
interactions  in  their  environment  may  have  played  a larger  role  in  prompting  them  to  engage  in  new  behaviors. 

Limitations 

The  CHAT  approach,  as  opposed  to  a traditional  cognitive  research  paradigm  to  study  the  impact  of 
IDEAS  on  these  educators  provided  a much  richer  understanding  of  the  interactions  among  teachers,  new 
content,  and  their  environment  as  they  made  change  in  teaching  methods  and  began  to  adopted  new 
technologies  and  resources  into  their  teaching  practices.  Given  this  was  a pilot  test  of  a new  research  and 
analysis  methodology  findings  must  be  interpreted  cautiously.  Only  two  subjects  were  investigated  and 
Davydov’s  (1999)  stages  were  not  followed  completely.  As  a result,  additional  analysis  will  be  conducted  to 
include  the  resulting  lesson  plans  as  data  points,  all  of  the  participating  educators  will  be  included  in  the  final 
analysis,  inquiries  will  be  made  to  other  educators  who  were  not  directly  involved  in  the  academy,  and 
significant  features  of  the  environment  will  be  further  explored.  More  observations  and  in-depth  semi-structure 
interviews  will  also  be  conducted. 

A Research  Model  Generated  from  the  CHAT  Case  Study 

CHAT  research  paradigm  provides  enlighten  researchers  with  a more  holistic  method  for  exploring 
technology  integration  efforts.  This  requires  a shared  understanding  of  the  character  and  history  of  the  subject, 
the  object  unto  which  the  subject  is  attempting  or  required  to  reach,  the  characteristics  of  the  surrounding 
community,  and  the  technology/tools  available  to  the  subject  etc.  A research  model  based  on  CHAT  paradigm 


499 


and  this  research  demonstrates  how  to  explore  the  research  context  and  review  the  literature  to  focus  on  the 
interaction  of  human  activity  and  the  whole  of  the  individual’s  mentality  as  they  interact  with  a relevant 
environmental  context  and  ultimately  finalize  appropriate  research  questions  and  methodologies.  This  model 
also  demonstrates  how  a research  conceptual  framework  can  be  established  to  integrate  research  questions,  data 
collection,  analysis , and  reporting. 

Stagel:  Research  Context  Exploration.  The  CHAT  framework  reveals  the  holist  research  context.  It  helps 
researchers  to  conceptualize  the  complexity  of  the  research  context  in  terns  of  the  characteristics  of  the 
technology  integration  activities,  the  factors  that  affect  change,  and  the  interactions  among  factors.  In  this  stage, 
researchers  need  to  explore  (1)  social,  historical  and  technology  attribute  characteristic  of  the  target  population; 
(2)  the  environment  in  which  the  target  population  operates  including  community,  rules,  and  division  of  labor; 
and  (3)  the  goals  the  target  population  is  trying  to  reach.  Identifying  the  characteristics  and  issues  surrounding 
these  factors  provides  a research  context. 

Stage  2:  Literature  review.  Based  on  understanding  the  general  characteristic  of  the  research  context, 
researchers  can  begin  to  review  the  literature  regarding  (1)  the  relationship  between  the  characteristics  of  the 
target  population  and  the  technology  use  (2)  the  interactions  among  target  population  and  environmental 
factors,  and  other  relevant  relationships.  The  review  should  focus  on  such  relationships  and  methods  used  to 
capture  data  and  understand  the  complexities  of  the  similar  environments. 

Stage  3:  Define  Research  questions.  Based  on  the  results  of  the  stagel  and  2,  researchers  further  define  the 
impact  of  technology  interaction  and  narrow  down  research  questions  to  “what”  and  “how.” 

Stage  4:  Establish  research  conceptual  framework.  A specific  research  conceptual  framework  is  then 
developed  based  on  the  research  questions  guided  by  an  analysis  of  the  activity  structure.  Elements  of  the 
research  process  such  as  research  design,  measures,  data  collection  instruments,  data  analysis  and  interpretation 
are  defined  based  on  current  understanding  of  the  activity  framework.  For  example,  in  the  aforementioned  case 
study  we  identified  (i)  academy  tools  (ii)  the  establishment  of  rules  of  engagement  and  other  factors  as  critical 
to  this  study.  Thus,  it  is  suggested  that  researchers  integrate  the  finding  of  previous  stages  to  establish  a specific 
research  conceptual  framework,  describing  the  research  purpose,  context  and  methodologies.  Refer  to  figure  2 
for  framework. 

Stage  5:  Data  collection.  The  CHAT  framework  is  then  used  to  design  appropriate  research  methods  and  select 
an  appropriate  sample  of  representative  participants  that  account  for  the  attributes  of  populations  and  the 
contexts  in  which  they  are  engaged.  In  the  IDEAS  case  study  we  selected  teachers  that  were  representative  of 
the  target  audience  for  the  technology  integration  academy,  who  had  a variety  of  experiences  and  different 
levels  of  attitudes  toward  the  use  of  technology  in  the  classroom  and  were  currently  engaged  in  a variety  of 
teaching  contexts.  The  variables  for  study,  situated  within  the  research  framework,  also  need  to  be  defined  and 
the  data  collection  methods  established  to  effectively  view  the  activity  under  investigation  from  multiple 
perspectives.  In  the  IDEAS  project  data  were  collected  at  time  intervals  to  investigate  changes  based  on  an 
intervention.  Repeated  measures  methods  were  used  to  collect  both  quantitative  and  qualitative  data,  either  of 
which  could  have  been  analyzed  to  show  change  over  time,  and  both  were  used  to  show  change  and  interactions 
at  different  times  throughout  the  project. 

Stage  6:  Data  analysis.  Research  conducted  based  on  CHAT  framework  yield  richer  data,  which  provides  more 
comprehensive  results  and  a stronger  “feeling”  of  understanding  of  the  changes  within  the  activity.  The 
researchers  are  better  able  to  picture  a holistic  view  of  the  changes  by  investigating  the  relationships  across 
different  analysis  results , using  a variety  of  data.  The  profiles  created  for  each  educator,  in  this  case  study, 
presented  a picture  of  how  the  individuals  changed  over  time.  Such  data  were  also  used  to  compare  and  contrast 
the  change  across  several  individuals  engaged  in  the  activities.  In  addition,  as  the  change  is  dynamic,  having 
measures  of  the  multiple  factors  within  the  activity  added  to  ability  to  identify  causal  and  intervening  variables. 
Such  rich  data  however  requires  strong  data  analysis  skills  that  inform  the  interpretation  process. 

Stage  7:  Data  interpretation  and  report.  Technology  integration  activities  are  complex  in  nature  and  unpacking 
the  factors  that  encourage  changes,  temporary  and  sustained,  is  a difficult  process.  Having  richer  data  that 
provides  insight  on  the  foundational  elements  of  an  activity,  e.g.,  subject,  object,  tools,  and  community, 
provides  perspective  on  the  whole  activity.  Gaining  insight  in  a comprehensive  and  understandable  manner  is 


500 


still  a matter  of  skillfully  applying  analysis  and  interpretation  techniques,  not  unlike  complex  quantitative  or 
qualitative  data  analysis  . It  is  recommended  that  interpreting  the  results  begins  with  basic  analysis  of  changes 
between  start  and  finish  state  of  the  subject  moving  toward  a goal  such  as  changes  in  attitudes,  or  measures  of 
learning.  The  analysis  then  continues  by  looking  at  the  relationships  between  the  other  elements  within  the 
activity.  Either,  or  both,  may  be  using  quantitative  or  qualitative  techniques,  however  the  key  is  to  work  through 
the  complexities  of  the  entire  activity  and  the  factors  that  influence  the  ebbs  and  tides  of  changes.  Therefore,  it 
is  suggested  that  interpretation  begins  in  simple  terms  and  eventually  considers  the  wholeness  of  the  activity 
and  the  interactions  that  occur  among  the  different  elements . 

Conclusion 

Traditional  cognitive  research  approaches  to  technology  integration  research  do  provide  valuable 
information,  but  generally  lack  the  robustness  to  fully  understand  the  dynamics  of  this  activity.  Conducting  such 
research  using  a CHAT  strategy  helped  to  reveal  technology  integration  activity’s  content,  structure, 
organization  and  fundamental  characteristics  as  they  exist  within  the  training  and  classroom  context.  Although 
much  more  complex  design,  such  an  analysis  helped  shed  light  on  the  complexities  of  technologies  use  to 
enhance  teaching  and  learning  and  how  such  tools  are  adopted  to  meet  instructional  needs  of  educators. 
Developing  this  understanding  will  help  in  the  development  of  more  comprehensive  research  and  evaluation 
methodologies  as  well  as  technology  integration  training  and  strategies. 
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Abstract 

In  the  information  age,  Educational  Technologists  exist  in  complex,  codependent  organizations  where  the 
information  age  is  demanding  changes  to  our  theory  and  praxis  (Reigeluth,  2001).  In  this  paper,  the  author 
argues  that  advanced  leadership  theory  and  practice  can  be  used  to  characterize  and  design  educational 
technology  R&D,  along  with  praxis,  to  account  for  the  relatively  incredible  potential  of  this  field  to  add  to  the 
social  capital  of  our  institutions,  states  and  nations.  First,  the  paper  presents  the  need  for  specialized 
educational  technology  leadership  epistemology.  Next,  the  macro  concept  of  social  capital  is  presented  as  a 
model  for  describing  high  level  contributions  from  the  field,  focusing  on  identifying  key  elements  educational 
technology  leaders  need  to  understand.  Finally,  high  capacity  network  characteristics  and  examples  from 
research  are  presented  to  inform  ET  leaders  about  what  it  takes  to  create  and  sustain  the  necessary  high 
capacity  leadership  network  in  the  information  age. 

Introduction:  A Need  For  Leadership  Knowledge  in  the  Educational  Technology  Field 

Traditionally,  the  assets  of  a commercial  organization  were  measured  in  terms  of  plant  and  property  as 
hard  assets  that  were  managed  as  relatively  stable,  reasonably  predictable  commodities.  The  most  highly  valued 
companies  were  the  ones  that  kept  production  turn-around  time  to  a minimum,  balanced  stock  and  sales,  and  got 
the  best  return  on  financial  reserves  to  provide  a prudent  mixture  of  debt  and  equity  for  the  future.  In  the  new 
“information  age”  or  “knowledge”  economy,  the  most  highly  prized  companies  are  the  ones  that  manage 
intangible  assets,  such  as  an  ability  to  generate  value,  to  create  and  maintain  social  and  intellectual  capital  for 
example,  rather  than  to  only  account  for  hard  assets  (Kelly,  2004;  Fullan,  2000). 

This  newer  commercial  paradigm  has  arguably  not  yet  affected  today’s  education  institutions  - but  it  is 
a widespread  condition  in  industry  (Senge,  2000).  Because  schooling  is  a society  sponsored  activity,  our 
intangible  asset  leadership  is,  indirectly,  becoming  prevalent  and  important  as  a leadership  example  across 
education  and  industry  training  sectors  (Bennis  et  al,  2003;  Helliwell,  2002;  Bolman  & Deal,  2000). 

Educational  technologists  work  in  both  industry  and  schooling.  Are  we  prepared,  in  ET  theory  and  in  practice, 
to  answer  to  this  critique  from  an  educational  technology  (ET)  field  perspective?  A common  premise  behind  the 
government  (grant  offering)  policy  thinking  may  be  important:  If  public  and  private  organizations  can  not 
organize  themselves  to  create  appreciating  (positive)  social  capital,  the  projects  completed  may  have  less  value 
to  society.  Are  we,  as  educational  technology  leaders,  knowledgeable  and  prepared  for  these  changing 
conditions?  It  is  one  thing  to  be  a good  instructional  designer  or  developer  - it  is  another  to  understand  the 
impact  of  the  projects  we  lead  in  institutions,  partnerships,  governments  and  society.  This  author  posits  that 
educational  technologists  have  a tremendous  contribution  to  offer  in  terms  of  social  capital  creation 
(appreciation),  and  that  we  need  to  learn  new  the  ways  of  modeling  and  managing  our  ET  work  within  a this 
larger  policy  sphere. 

Government  and  industry  view  education  as  a significant  potential  contributor  to  social  capital 
generation  (Woolcock  and  Narayan,  2000),  so  educational  technologists  are  directly  concerned.  This  is  not  only 
because  we  practice  in  both  industry  and  education  sectors,  but  also  because  we  are  often  responsible  for  large, 
expensive  public  projects  that  involve  a lot  of  human  and  social  capital.  It  is  possible  that  under  these  conditions 
that  without  significant  change,  our  field  “could  be  relegated  to  its  backwaters”,  and  that  inertia  could  influence 
whether  we  as  individuals  are  successful  in  our  careers  or  find  ourselves  progressively  less  effective  - much  as 
the  old  tightly  bureaucratic  firms  can  no  longer  handle  information  markets  and  much  change  (Fullan,  2000). 

ET  senior  scholar  Charles  Reigeluth  (2001)  lists  several  organizational  factors  for  educational  technologists  to 
consider  as  we  adapt.  As  this  paper  will  demonstrate,  the  macro  concepts  of  social  capital  and  networked 
organizations  make  reasonable  new  models  for  understanding  and  informing  educational  technology  leadership 
to  include  finite  characteristics  such  as  autonomy  accountability,  cooperative  relationship,  networking  and 
process  oriented  approaches  necessary  for  the  organization  of  the  future  (Dickson  et  al,  2003;  Reigeluth,  2001). 
But  it  is  not  enough  to  know  the  conditions  and  constraints  organizations  offer  us  today  - we  must  know  more 
about  how  to  lead  educational  technology  both  as  a field,  and  in  practice,  in  our  increasingly  interdependent 
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and  partnered  education,  government  and  corporate  settings  (LaGrange,  2004;  Hargreaves,  2001;  Wheatley, 

2000). 

In  this  brief  paper  the  author  introduces  the  imperative  for  educational  technology  leaders  to  consider 
social  and  intellectual  capital  as  macro  constructs  in  our  leadership,  along  with  definitions  of  the  terms.  The 
important  concepts  of  networks  and  network  leadership  are  explored,  with  some  examples  of  high,  medium  and 
low  capacity  technology  leadership  networks  offered  from  recent  research  findings.  The  paper  concludes  with 
an  argument  to  place  educational  technology  in  the  ‘dead  center’  of  the  recent  trend  to  make  social  science 
matter  today  in  high  policy  circles  (Judge,  2004).  A summary  of  the  organization  and  structural  characteristics 
inherent  in  high  capacity  social  capital  generating  networks  is  then  offered. 

Social  Capital:  A Macro  Level  Concept  for  ET  Leadership  Today 

The  late  American  sociologist,  James  Coleman  (1988)  argued  that  social  capital  consists  of  those 
aspects  of  social  structures  and  systems  that  facilitate  the  actions  of  actors  within  a structure.  As  such,  social 
capital  can  be  used  as  an  important  model  allowing  an  aggregate  measure  or  descriptor  of  complex  systems  of 
educational  leaders  who  work  together  via  network  structures  to  improve  education  (Kelly,  2004).  The  concept 
offers  an  encompassing  description  of  agencies,  governments  or  entire  sectors  because  the  unit  of  study  can  be 
approached  at  either  the  micro  (individual),  rneso  (institutional)  or  macro  (regional,  global,  economic)  levels 
(Hall,  1994).  Social  Capital  is  a macro  concept  that  can  be  defined  in  two  ways.  First,  by  its  structural 
components  (as  networks  of  interacting  individuals  or  organizations)  and  second  by  its  cultural  components 
(Bourdieu,  1986;  Judge,  2004).  The  cultural  components  of  social  capital  have  long  been  the  focus  of  social 
science  work  on  social  networks,  concentrating  on  actor  (leader)  obligations  and  expectations,  trust,  information 
potential,  norms  and  effective  sanctions,  authority  relations,  and  appropriate  social  organizations.  This  kind  of 
social  capital  cultural  research  is  heavily  dependent  on  actor  contact  time  and  other  measures  using  social 
network  theory  - an  intuitively  pleasing  idea  which  slipped  into  a bit  of  heuristic  confusion  in  the  1990s 
(Rhodes,  1996). 

Describing  social  capital  as  structural  networks  of  people  organizing  their  work,  Bourdieu  (1986) 
classified  this  kind  of  capital  as  an  asset  found  in  networks  where  three  types  of  social  capital  are  known  to  exist 
in  pseudo-institutional  environments.  Such  network  types  are:  bonding  (within  homogeneous  groups),  bridging 
(networks  where  actor  relations  cut  across  diverse  social  cleavages  horizontally)  and  linking  networks  (where 
ties  are  formed  between  different  strata  of  status,  influence  or  wealth).  Describing  intangible  assets  in  this  way, 
a process,  involves  studying  flows  of  information  and  the  types  of  relationships  existent  in  both  institutional  and 
political  terms.  Recent  policy  network  scholars  have  developed  a set  of  network  characteristics  that  allow  us  a 
model  and  describe  the  types  of  relations  among  many  kinds  of  entities  a wide  array  of  contexts  (Judge,  2004; 
Bourdieu,  1986).  Strong  relational  links  in  these  networks  generally  mean  a highly  capable  system  for 
generating  social  capital,  but  weaker  links  and  bonds  (like  those  used  by  job  seeking  individuals)  can  also  mean 
much  more  social  capital  for  individuals  who  must  work  across  groups  and  fields,  for  example  (Granovetter, 
1978),  so  the  network  concept  is  not  quite  as  intuitive  as  it  sounds. 

By  applying  more  advanced  political  science  derived  constructs  of  policy  networks  to  describe  the 
relational  (structural)  networks  in  organizations  (and  between  them),  the  process  of  interest  organization  (what 
matters  to  who,  and  how  they  respond  to  pressing  issues  within  influence  networks)  can  be  studied  to  create  an 
encompassing  and  discriminating  method  for  understanding  social  networks  because  structures  with  specific 
taxonomies,  capacities  and  autonomies  can  be  identified  by  how  folks  organize  their  interests  (Atkinson,  1996; 
Coleman,  1998).  In  2003,  Kowch  modified  policy  network  theory  - an  advancement  on  earlier  social  network 
theory  to  include  influence  and  power.  This  extension  allows  the  description  of  structural  social  capital  elements 
(networks)  more  precisely  at  the  instutitonal  level.  Kowch  studied  ducational  technology  leadership  (influence 
networks)  across  states,  governments  and  large  institutions  to  identify  network  philosophical,  leadership  style 
and  educational  technology  actor  ontologies  as  important  sets  of  variables  policy  leadership  systems  that 
emerged  across  several  universities,  for  example. 

In  his  book  “Bowling  Alone”,  Harvard’s  Robert  Putnam  revealed  a national  study  describing  and 
explaining  some  of  the  reasons  for  a steady  decline  of  social  capital  in  America  over  the  last  50  years,  in  social 
capital  in  America  (Putnam,  2000).  He  measured  social  capital  via  social  network  analysis,  including  a deep 
analyses  of  volunteerism  in  America.  He  defined  a sharp,  continuing  decline  in  American’s  participation  in 
professional  organizations  and  public  institutions  - a depreciation  in  social  capital.  Among  his  findings  are 
revelations  that  education  (capital)  systems  require  more  funding  to  increase  education  system  performance,  and 
that  more  children  (learners)  must  be  included  in  education  networks.  From  these  and  similar  findings  arise 
powerful  implications  and  models  for  the  leadership  of  any  educational  technology  R&D  in  both  public  and 
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private  settings. 

Educational  technologists  today  are  working  more  frequently  within  and  across  the  domains  of 
utilization,  design,  development,  management  theory  and  practice  simultaneously  in  complex  institutional 
settings  and  partnerships.  These  partnerships  are  often  formed  with  with  the  state  government  or  local 
governments,  where  a codependence  often  exists  between  the  institutions  working  together  on  ET  projects.  It 
seems  then,  that  a structural  or  network  approach  to  understanding  ET  leadership  could  be  very  helpful  as  a way 
of  interpreting  and  designing  information  age  ET  social  capital  generating  projects  in  a time  of  great  change  and 
organizational/political  complexity  (Reigeluth,  2001),  especially  when  codependent  relations  are  essential  to 
our  ever  increasing  project  complexity  and  size,  and  when  the  type  and  kind  of  network  (bridging,  bonding, 
linking)  may  need  to  be  carefully  designed  and  used  to  measure  our  successful  ET  contribution  to  social  capital. 
This  is  an  important  statement  when  governments  are  generating  policy  to  fund  and  to  promote  primarily  high 
social  capital  education  endeavors  (Woolcock,  2000). 

Effective  leadership  networks  create  a high  social  capital  generating  capacity.  For  example,  across- 
school  division  improvements  or  pan-institutional  corporate  training  through  ET  work  is  common  practice  for 
us  in  the  des  ign  and  development  we  do  in  the  field.  Can  we  account  for  this  value-added  intangible  asset? 
Distance  learning  has  increased  our  need  for  financial  capital  while  at  the  same  time  it  is  eleasing  us  from 
geographically  constrained  relations  - a factor  that  is  still  a condition  constraining  most  fields  in  education  . 
Think  of  this  kind  of  ET  asset  or  capital  from  a government  granting  agency  perspective.  Independent  of  the 
product  of  our  usual  project  processes,  can  educational  technologists  claim  increased  social  capital  generation 
because  of  the  processes  we  use  to  get  things  done?  I think  we  can.  This  is  because  we  link  all  sorts  of  experts 
and  people  in  meaningful  ways,  across  social  strata,  diverse  social  cleavages,  institutions  and  governments  - 
consistently.  If  a granting  agency  is  considering  the  linking,  bridging  and  bonding  characters  of  a project  as  it 
creates  links  between  and  across  organizations  or  entities,  we  among  most  educators  create,  design,  maintain 
and  yes,  provide  leadership  for  such  work  consistently.  Compared  to  a project  in  counseling  psychology  for 
example,  we  generate  far  more  social  capital,  by  definition.  We  rarely  document  that  contribution  in 
organizational  and  social  contexts. 

As  educational  technology  leaders,  there  exists  little  evidence  that  we  have  found  a way  to  describe  the 
benefit  (appreciation),  in  terms  of  our  organizations,  of  these  intangible  systems  when  they  work.  Yet  there  is 
little  doubt  that  these  systems  add  to  the  capital  of  the  organization,  state  or  nation  (Kowch,  2003).  If  we  can 
describe  these  network  processes,  we  should  also  move  to  expand  theory  so  that  we  can  design  and  be 
accountable  for  the  social  capital  generated  by  such  capital  generating  networks.  Figure  1 demonstrates  the 
three  types  of  (structural)  social  capital  developed  by  networks. 


Type  of  Social  Capital  (Network)  found  in  institutional  arrangements 
(Kowch  after  Bourdieu,  2003) 

Bonding 

- Relations  within  homogeneous  groups 

- Strong  ties  mean  stronger  capital  (Putnam,  1998,  Woolcock, 
2004) 

- Example:  Teachers  Associations 

Bridging 

- Relations  cut  across  diverse  social  cleavages 

• Horizontally  - key  to  exchanging  new  info 
- Useful  for  information  diffusion 

• Weak  ties  mean  low  capital,  and  be  an  advantage  for 
innovation  / or  organizational  change  (capacity) 

- Example:  A distance  education  project  consortia,  led  by  a 
university  organized  to  design  and  deliver  technology  leadership 
education  to  both  school  principals  and  home  school  parents  in 
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the  Australian  outback. 


Linking 

- Ties  between  different  strata  of  wealth  and  status  (Woolcock, 

2003) 

• V ertically  - key  to  linking  formal  institutions 

- Ties  between  different  strata  of  influence  (Kowch, 

2003). 

- Example:  Educational  Technology  Leaders  designing,  creating 
and  implementing  state  level  distributed  education  procedures  / 
policy,  and  expanding  the  partnership  to  include  European 
states,  universities  and  schools  to  do  the  same  thing. 

- Example:  ET  leaders  in  multinational  corporations  leading 
consortia  development  of  new  satellite  based  education 
programs  across  the  globe  in  several  countries. 

Figure  1:  Types  of  social  capital  with  Educational  Technology  Field  examples 

Leading  Educational  Technology  via  Network  Leadership:  A Meso  Level  Concept 

Developing  and  leading  in  the  disaggregated  or  neo-institutional  organization  means  knowing  more 
about  influence  networks  that  get  things  done  (policy  networks)  and  the  larger  communities  that  these  networks 
exist  within.  In  this  section,  policy  networks  are  presented  as  embedded  social  capital  generating  entities  within 
larger  communities,  all  spanning  organizations  and  / or  governments.  This  section  begins  with  policy  and 
network  definitions,  and  closes  with  a description  of  the  key  variables  in  a high  capacity  network  so  that 
educational  technology  leaders  can  ponder  those  parameters  - with  an  eye  to  generating  robust  organizations  as 
they  lead  ET  developments. 

Policy  Network 

Originally,  policy  network  analysis  was  created  as  a response  to  the  limitations  of  public  choice 
(competitive)  hierarchical  policy  and  organization  models  in  efforts  to  better  explain  how  people  organize  their 
interests  in  order  to  react  and  to  get  policy  (making)  work  done  (Wilks  & Wright,  1987).  Because  older  linear, 
rational  choice  institution-based  frameworks  were  found  lacking  in  their  capacity  to  describe  the  reality  of 
government  / industry  relations  in  an  increasingly  partnered  government  and  institutional  (information  age) 
world,  a new  frame  of  analysis  was  needed  (Lane  1995).  Rather  than  conceptualizing  government  as  a 
benevolent  distributor  of  resources  to  competitors,  network  policy  analysts  accepted  that  government/industry 
or  sectoral  partnerships/relations  today  are  the  norm  (Pal,  1997).  The  utility  of  traditional  public  choice,  public 
policy  or  public  administration  models  to  describe  how  such  interwoven,  codependent  governments,  industries 
and  sectors  organize  their  interests  was  deemed  deficient,  so  policy  network  research  emerged  (Atkinson  & 
Coleman,  1996).  Policy  issues  or  problems  are  no  different  than  strategic  or  operational  problems  in  an 
organization,  for  policy  is  defined  by  the  policy  network  framework  theorists  as  a process  - a reaction  (or 
inaction)  to  an  issue  by  a group  of  people  (Coleman  & Skogstad,  1990)  who  coalesce  from  a larger  constellation 
of  individuals  (a  policy  community)  to  act  on  the  problem  (to  solve  it).  Because  networks  are  organizations,  and 
since  organizations  are  knowledge  (Van  Wijk,  2003),  this  author  suggests  that  the  process  of  interest 
organization  is  not  far  different  from  knowledge  management  (KM)  (knowledge  transfer  and  flow),  and  that 
policy  network  conceptual  frameworks  are  very  similar  to  knowledge  management  networks.  In  other  words, 
knowledge  management  theorists  may  use  policy  network  capacity  models  to  define  (or  design)  high  capacity 
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knowledge  networks.  For  example,  consider  the  ET  leader  who  must  negotiate  copyright  protocols  for  a 
distance  project  that  involves  information  that  will  be  generated  across  several  universities  and  states.  The 
complexity  of  people  and  activity  in  a large  project  is  huge,  but  if,  as  leader,  you  understand  what  moves  people 
to  work  in  vibrant,  codependent  networks  across  governments  and  institutions,  that  complexity  is  rendered 
simple  if  you  consider  the  nature  of  high  capacity  networks  (at  the  end  of  this  section).  Older  bureaucratic  and 
functional  leadership  requires  a much  more  complex,  incremental  approach  - today,  network  architects  must 
understand  KM  too. 

So  we  can  consider  the  ‘organization’  of  a big  project  to  be  a collection  of  actors  who  are  reacting  to  a 
problem  or  challenge  as  they  organize  what  they  think  is  important  (about  the  problem).  We  see  study  them  as 
they  subsequently  go  about  the  process  of  developing  tangible,  tactical  or  strategic  responses  to  that  problem 
(solutions).  The  exact  interactions  are  not  as  important  in  a study  of  change-capable  organization  compared  to 
the  importance  of  knowing  the  type  of  process  this  group  or  network  exhibits.  In  the  neo  institutional  or 
codependent  groups  of  today.  Institutional  actors  do  not  identify,  rank  and  solve  problems  in  isolation,  and  they 
do  not  necessarily  organize  themeslves  hierarchical  or  bureaucratic  structures.  They  can  coalesce  or  come 
together  from  a constellation  (community)  of  actors  who  have  similar  interests  - in  effect,  forming  a 
knowledgeable  group  of  actors  who  create  a network  to  get  work  done  (Alvesson,  1996;  Garcea,  1997).  Do  we 
organize  our  teams  with  this  knowledge,  considering  these  organizational  factors  in  educational  technology 
field? 

To  demonstrate:  If  water  rights  become  an  issue  (or  problem)  for  a collection  of  farmers,  industries  and 
governments  in  a region,  you  can  safely  bet  that  a collection  of  interested  individuals  will  form  (with 
management  or  perhaps  without  it)  to  generate  a response  and  to  push  for  solutions.  What  has  been  described 
here  is  the  identification  and  ranking  of  a policy  issue  being  across  institutions,  and  then  organized  by  people 
within  a network  or  pattern  of  relations.  These  people  may  (or  may  not)  successfully  create  a solution  to  the 
problem  by  working  with  actors  across  departments  in  a company,  from  various  farms,  and  from  across  the 
government  agency  responsible  for  the  sector,  depending  on  their  motivation,  and  common  interest,  and  on  their 
collective  capacity  to  organize  their  interests  in  order  to  act  (to  solve  the  problem).  This  is  complex  activity  to 
model,  and  network  analysis  allows  us  to  render  these  types  of  emergent,  pressing  and  sometimes  quick- 
forming pan-institutional  processes  simply  (Lane,  2000).  There  is  value  rendering  such  complexity  simple.  In  a 
business  or  corporate  setting,  the  same  pan-institutional  issues  and  resulting  activities  (processes)  can  occur 
when  two  companies  merge.  For  example,  say  two  different  compensation  schedules  need  to  be  negotiated 
because  of  a new  partnership  formed  by  merger.  Some  people  in  both  the  new  organization  will  be  motivated  to 
achieve  a certain  solution  over  other  possible  solutions,  and  they  will  be  motivated  to  work  together  to  define, 
rank  and  organize  their  key  interests  or  issues,  to  exchange  knowledge,  and  to  work  to  create  a solution  or 
response.  This  will  occur  no  matter  how  far  flung  the  various  divisions  or  regions  are  geographically  if  the  issue 
is  important  enough  (Coleman  & Skogstad,  1990).  Because  labor  laws  and  government  regulations  may  differ 
in  different  states,  government  may  be  involved  in  the  solution  as  well.  The  capacity  of  the  network  to  find  and 
organize  its  interests  will  be  important  for  all  concerned,  so  that  the  process  serves  some  end  or  solution  (work). 
Wise  Educational  Technology  leaders  in  the  near  future  should  be  able  to  craft  such  a network  for  success,  or  at 
lest  to  describe  the  parameters  that  characterize  such  a high  capacity  network. 

In  the  next  section,  this  paper  demonstrates,  briefly,  the  findings  from  a study  on  how  three  universities 
in  two  states  organized  their  interests  to  set  the  educational  technology  (issue)  directions,  investing  millions  of 
dollars  while  indirectly  affecting  hundreds  of  thousands  of  students  and  many  faculties.  Policy  network  study  is 
a study  about  what  issues  draw  people  to  an  action  network,  about  why  they  were  drawn  to  the  issue,  and  about 
how  they  organize  their  interests,  in  patterns  or  structures  (called  networks)  to  make  things  happen.  It  is  a study 
of  the  how  of  networks,  not  only  of  the  whom  and  what  of  networks.  In  policy  network  study,  individuals  or 
actors  are  analyzed  at  the  micro  level,  and  the  network  (pattern)  is  analyzed  at  the  rneso  or  neo-institutional 
level.  Because  interest  (knowledge)  organization  processes  are  studied  and  interpreted  using  an  extension  of  the 
functionalist  (descriptor  only)  policy  network  canon  in  this  process,  a post  hoc  analysis  can  interpret  the  nature 
of  the  interest  organization  process  itself  - and  that  information  can  inform  network  design  for  ET  leaders  who 
are  engaged  in  creating  high  capacity  or  high  social  capital  generating  networks  in  similar  situations  (Kowch, 
2003). 

Viewed  from  an  ontological  perspective,  the  policy  network  canon  still  provides  mostly  functionalist 
sociology  and  organization  theory  frameworks  too,  but  leaders  need  to  consider  the  way  actors  view 
organization  (and  leadership)  processes  as  well.,  as  the  author  found  that  leaders  who  see  organizing  as  a 
bureaucratic  process  are  essentially  left  out  of  influence  networks  that  are  fluid  and  post  bureaucratic.  This  is 
why  very  few  educational  technologists  were  nominated  to  the  educational  technology  influence  network  in  the 
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Western  Canada  study  (only  two  were  nominated  from  a possible  sample  of  over  60).  When  designing  high 
capacity  networks  (high  capital  networks)  the  concept  of  neo-institutionalism  is  helpful  in  breaking  the  bounded 
rationality  proposed  by  internal/external  institutional  analysis  (Lane,  1995).  The  neo  institutional  construct  is 
particularly  useful  when  designing  or  interpreting  partnered  organization  processes  (networks)  that  might  not 
function  entirely  bureaucratically,  hierarchically  or  even  as  closed  systems  (Kowch,  2003).  Neo-institutionalism 
is  the  condition  where,  in  a disaggregated  state  (where  government  and  industry  share  in  responding  to  issues), 
institutions  have  considerable  autonomy  to  organize  interests  and  to  create  strategies  for  problem  solving.  That 
autonomy  is  of  course  a function  of  the  capacity  of  the  institution  to  exchange  ideas,  and  upon  the  pattern  of 
relations  by  they  choose  to  exchange  the  flow  of  ideas  (Atkinson  & Coleman,  1996).  The  basic  unit  of  analysis 
at  this  institutional  (meso)  level  is  the  pattern  of  relations  between  individuals  who  depend,  to  varying  degrees, 
on  each  other  to  exchange  (and  to  generate)  information  while  they  organize  their  (main)  interests  or  problems 
(Howlett  & Ramesh,  1995).  Neo  institutional  interpretive  frameworks  allow  us  to  characterize  both  internal  and 
external  (pan  institutional  and  community  invested)  representations  as  one  network  and  the  concept  maps  well 
with  the  increasingly  necessary  flexible,  recursive  or  constructivist  ontologies  found  in  ET  praxis  (Salomon, 
2000). 

Characterizing  high  capacity  networks 

According  to  Coleman  & Skogstad  (1990),  high  capacity  issue  organizing  policy  network  actors 
possess  the  following  characteristics: 

(1)  a clear  concept  of  role  in  the  process  or  organizing  things, 

(2)  a supporting  value  system  (supporting  the  network  defined  goal) 

(3)  a unique,  professional  ethos, 

(4)  an  ability  to  generate  information  to  answer  unanswerable  questions, 

(5)  an  ability  to  maintain  cohesion  within  the  network 

(6)  an  ability  to  organize  and  manage  complextasks,  leading  to  a work  output  (result),  and 

(7)  the  ability  to  rise  above  the  (near  term)  self  interest  of  the  group  (network). 

In  addition,  Garcea  (1997)  notes  that  high  capacity  actors  have  three  characteristics  that  affect  the 
capacity  of  networks  to  get  things  done:  (1)  interests;  (2)  institutional  contexts  (programmatic  or  political,  & 
managerial  and  financial  management  capacity)  and  (3)  ideologies  or  ontologies.  Subjective  or  objective 
ideologies  or  ontological  stances  to  the  organization  task  are  important  to  know,  as  they  are  important  factors  in 
the  interpretation  of  the  capacity  of  the  response  networks  (Kowch,  2003).  By  linking  management  models 
describing  the  capacity  of  networks  to  get  things  done  (and  to  handle  change),  Kowch  borrows  from  Ibarra 
(1992)  to  describe  the  dynamic  potential  of  policy  networks  to  get  issue  organization  (knowledge)  work  done. 
Ibarra’s  model  was  based  on  previous  work  in  social  networks  (Granovetter,  1973)  and  Kowch  used  it  to  add 
analytical  validity  at  the  meso  level  (network)  analysis  of  policy  networks  to  provide  a complimentary 
description  of  loose  or  tight  ties.  Both  methods  yielded  the  same  network  descriptions  for  network  change 
capacity  and  innovation  capacity.  So  these  are  the  criteria  by  which  the  process  of  organization,  evidenced  by 
policy  networks,  is  characterized.  The  result,  to  long  to  mention  here,  is  a method  for  characterizing  the  change 
capacity  of  networks  that  generate  positive  social  capital  in  a neo-institutional  or  complex  organizational 
setting.  The  type  of  organizing  the  network  does  (i.e.  pluralist,  corporatist,  concertist,  statist)  can  also  be 
identified  by  doing  an  autonomy  analysis  (Lindquist,  1996).  In  this  paper,  only  the  network  capacity 
determination  will  be  demonstrated  for  parsimony  reasons.  From  the  previous  literature,  it  is  clear  that  high 
capacity  networks  also  contribute  to  high  social  capital  generation  networks.  From  descriptions  and  analyses  of 
these  three  policy  network  case  studies,  performed  at  the  micro  (actor),  meso  (network)  and  macro 
(environment)  levels,  the  author  then  presents  findings  and  a more  detailed  analysis  of  policy  issue  (knowledge) 
organization  networks  at  the  institutional  level. 

High  and  Low  Capacity  Educational  Technology  Leadership  Network  Examples  (High 

and  Low  Social  Capital  Generators) 


A Low  Capacity  ET  Leadership  Network  Case  - Too  Many  Interests,  Some  Codependence  in  a Workflow 
Network 

In  earlier  research  (Kowch,  2003)  a low  capacity  case  ET  leadership  network  describes  a closed  system 
of  faculty  and  administrators  who  came  together  with  two  issues  or  fundamental  motivations  to  organize  in 
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mind  . This  caused  a cleavage  to  appear  in  the  network  diagram  (Figure  2).  One  low  capacity  cleavage  had 
weak  ties  to  the  other  interest  group,  and  collected  because  of  self  interest  in  distance  education.  The  higher 
capacity  cleavage  (a  larger  group)  came  together  to  solve  problems  related  to  technology  because  they  believed 
the  institution  required  a progressive  image.  Both  groups  were  unsure  of  their  role  in  the  network  as  policy 
makers,  while  all  actors  exhibited  a supporting  ethos  and  value  system  to  serve  students  and  the  institution.  The 
two  interests  in  the  influence  network  therefore  decreased  the  cohesion  and  the  organization  capacity  of  the 
network  to  get  policy  done  or  to  prioritize  what  (knowledge)  mattered  most  (and  they  readily  admitted  this  fact). 


Lower  Capacity 


Cleavage  / Distance  Higher  Capacity 


Figure  2:  Low  Capacity  Educational  Technology  Leader  Network  Case 

The  “distance  cleavage”  emerged  as  a subsystem,  and  evidenced  only  a weak  connection  to  the 
government  through  the  other  cleavage.  All  actors  in  the  case  were  found  to  have  a strongly  bureaucratic  or 
objectivist  organization  ontology,  and  most  actors  preferred  to  submit  decisions  to  their  respective  committees 
(85%  of  the  members  sat  on  each  other’s  committees).  Most  actors  knew  that  the  super  ordinate  committees 
they  chose  to  send  policy  creation  (work)  “up”  to  had  no  funds  or  policy  instruments,  and  likely  would  not  pass 
the  recommendations.  As  such,  this  is  a hierarchical  work  flow  network  that  is  tightly  knit,  impeding  innovation 
and  flexibility  to  respond  to  challenges  (Ibarra,  1992).  With  weak  ties  to  a government  member  who  also  looked 
to  the  committee  for  interest  organization,  this  network  was  classified  as  a pressure  pluralist  organization,  where 
both  actors  and  cleavages  created  a low  capacity  issue  organizing  network  with  low  social  capital  generation 
capability. 

A High  Capacity  ET  Leadership  Network  Case  - One  Interest,  High  Codependence,  Loose  Ties  in  an 
Non-  Workflow  (non  bureaucratic)  Network. 

Figure  3 depicts  the  structure  of  the  “Calliope”  University  case,  which  among  the  three  cases  showed 
the  highest  capacity  to  organize  interests  across  institutional  boundaries.  In  this  macro  environment,  the 
government  had  strategic  plans  for  the  universities,  and  required  the  universities  to  generate  plans  that  aligned 
with  government  plans  about  education  and  technology  - and  the  government  had  in  place  funded  policy 
instruments  (grants)  targeted  at  anyone  in  the  university  system,  so  the  policy  (macro)  environment  was  far 
more  organized  than  in  the  other  two  cases,  where  no  similar  government  plans,  policy  or  alignments  with  the 
institutions  was  evident. 

ET  Leadership  Network  Motivation:  Overall,  the  reason  for  people  coming  to  this  to  work  out  the 
problem  responses  was  found  to  be  a desire  to  increased  market  share  for  the  institution,  and  every  member 
indicated  this  one  issue  or  interest  driving  their  (network)  organization.  Though  holding  a predominantly 
determinist  view  of  education  technology,  these  people  come  together  to  set  policy  based  on  the  generally 
understood  idea  that  technology  will  give  a market  edge  to  the  university.  They  came  up  with  this  understanding 
as  a group,  but  held  the  ideal  individually  as  well,  and  actors  came  to  this  network  from  across  many  faculties, 
government  agencies  and  administrative  departments.  A key  interest  or  common  knowledge  is  what  holds  this 
network  together  - at  the  table  so  to  speak.  Everyone  understands  the  issue  and  has  “bought  into  it”. 

ET  Leadership  Network  Composition:  These  9 actors  have  a central  core  of  4 service  group  experts, 
with  the  others  being  physical  plant  people  and  executives  from  the  academic  and  administration  chambers,  also 
including  people  from  the  professorate  and  related  (higher  education)  government  senior  officials.  The  core  or 
non-workflow  (non  hierarchical  relation  - non  bureaucratic)  core  of  this  group  change  depending  on  the  ET 
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ET  Leadership  Network  Capacity:  In  this  network,  people  from  across  the  institution  share  one  interest 
with  the  government  and  they  have  the  same  joint  concern,  so  both  the  government  and  the  institutional  network 
demonstrate  a high  degree  of  autonomy  and  capacity  in  organizing  their  interest  (they  are  both  doing  what  they 
want  to  do,  depending  on  each  other  to  organize  the  one  main  issue  - market  share  gain),  and  they  both  are  able 
to  articulate  and  solve  the  problem  independently  (they  cooperate  in  that  process  as  a network,  without 
hierarchical  structure).  An  (aggregated)  but  more  detailed  description  of  the  key  capacity  characteristics  of  this 
network  follows,  presenting  findings  and  analysis  for  each  criterion  used  in  policy  network  capacity 
determination. 


State  Agency  Organization 


Low 


High 


Low 


Institutional 
Interest  Organization 
(Capacity) 


High 


Pressure  Pluralism 

State  1 
Direction  ; 

Corporatism 

Pluralism 

Figure  4\  Network  Type:  High  Capacity  Educational  Technology  Leader  Network  Case 


Though  it  is  beyond  the  scope  of  this  paper,  it  is  possible  to  characterize  complex  network  associations 
from  across  and  well  beyond  what  we  think  of  as  ‘the  organization’.  In  the  study  of  educational  technology 
leadership,  no  one  nominated  by  other  influential  actors  thought  that  they  were  leading  university  education 
technology  policy  - but  they  were.  By  studying  influential  leaders  who  did  not  know  all  the  participants  in  the 
network,  this  model  permits  us  to  understand  the  process  of  leadership,  not  only  the  product.  Deep  analysis  of 
the  leadership  and  organization  capacity  of  these  networks  is  possible  by  studying  how  they  organize  their 
interests  by  applying  advanced  organization  theory.  If  we  know  how  they  organize,  we  can  design  for  change. 

How  Can  Educational  Technologists  Create  and  Lead  High  Capacity  (Social  Capital  Generating) 
Educational  Technology  Efforts? 

In  this  paper,  the  author  argues  that  a new,  information  age  (Reigeluth,  2001),  neo  institutional 
approach  to  understanding  the  processes  whereby  our  organizations  deal  with  educational  technology  issues  is 
necessary.  As  an  educational  technology  field,  we  need  to  bring  our  management  domain  theory  up  to  today’s 
organizational  theory,  education  administration  and  knowledge  management  (business  school)  levels.  We  also 
need  to  consider  the  organizational  phenomenon  and  larger,  macro  issues  like  economics,  politics  and  history  in 
our  plans  to  make  our  work  sustainable  and  manageable  within  increasingly  complex  systems.  We  must  pay 
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particular  attention  to  the  kind  of  purpose  we  communicate  when  we  create  educational  technology  projects, 
and  plan  to  take  a distinct  leadership  role  in  the  generation  not  just  of  processes  for  implementation  and 
management,  but  in  the  design  and  guidance  of  bonding,  bridging  or  linking  organizational  networks  that 
accomplish  our  projects.  We  must  also  consider  policy  and  politic,  like  all  education  leaders  - finding  ways  to 
render  leadership  of  extremely  complex  phenomenon  simple,  as  we  can  with  the  social  capital  concept. 
Educational  Technology  matters,  and  our  community  and  field  offer  social  capital  generation  in  buckets  to  our 
governments  - perhaps  we  need  to  account  for  that  a bit  more. 

The  ET  field  is  increasingly  in  a position  to  provide  very  high  social  capital  generating  projects  from 
our  field,  and  we  need  to  be  able  to  account,  and  be  accountable  for  such  foresight,  change  and  leadership.  A 
good  beginning  is  the  inclusion  of  leadership  theory  and  philosophy  in  the  Educational  technology  graduate 
student  programming,  such  as  that  under  way  at  the  University  of  Calgary. 

Such  changes  may  indeed  reflect  the  recent  changes  suggested  by  social  scientist  at  large  in  a peal  to 
make  social  science  really  matter  again  (Flyvgjerg,  2001).  Flyvgjerg  suggest  that  as  social  scientists,  we  must 
move  along  among  the  three  intellectual  virtues  Aristotle  proposed,  from  episteme  (scientific  knowledge  of  our 
field)  to  techne  (pragmatic,  variable,  context-dependent  knowledge  of  our  field)  to  phronesis  (a  values  based, 
action  oriented  and  variable  rationality  less  instrumental  in  nature).  Indeed,  this  author  found  that  while  some 
educational  technology  leadership  networks  governing  over  200,000  faculty  and  students  in  two  states  had  high 
and  low  capacity  networks,  both  networks  maintained  a functionalist  (techne)  view  of  both  educational 
technology  in  education  and  educational  leadership.  By  broadening  the  field  to  include  leadership  and  political 
science  theory  (social  capital)  as  a model  for  describing  complex  issues,  patterns  and  relations  in  the 
information  era,  perhaps  we  can  strengthen  our  social  capital  capacity  to  account  for  our  ET  project  successes  in 
this  information  age. 


References 

Alvesson,  M.,  & Deetz,  S.  (1996).  Critical  theory  and  postmodernism  approaches  to  organizational  studies.  London, 
England:  Sage. 

Atkinson,  M.,  & Coleman,  W.  D.  (1996).  Policy  networks,  policy  communities  and  the  problems  of  governance. 
In  L.  Dobuzinskis,  M.  Howlett  & D.  Laycock.  (Eds.),  Policy  studies  in  Canada:  The  state  of  the  art  (pp. 
193-213).  Toronto,  ON:  University  of  Toronto  Press. 

Bennis,  W.,  Geary,  R„  Gery,  G.,  Burke,,  W.,  Juechter,  W.,  and  Tichy,  N.  (2003).  Dead  end  or  an  open  road? 

What  lies  ahead.  Training  and  development,  57,  (1),  (pp.  32-44). 

Carr,  N.  G.  (2003).  IT  doesn’t  matter.  Harvard  Business  Review,  81,  5-12. 

Coleman,  J.  S.  (1988)  Social  Capital  in  the  Creation  of  Human  Capital,  American  Journal  of  Sociology  , 94 
(Suppl.  95):  S95-S120 

Coleman,  W.,  & Skogstad,  G.  (1990).  Policy  communities  and  public  policy  in  Canada.  Toronto,  Ont:  Copp 
Clark. 

Dickson,  M.,  Gewirtz,  S.,  Halpin,  D.,  Power,  S and  Whitty,  G.  (2003).  Education  action  zones:  Model 

partnerships’,  in  Franklin,  B.,  Bloch,  M.  and  Popkewitz,  T.  (Eds.)  Educational  Partnerships  and  the 
State:  the  paradoxes  of  governing  schools,  children  and  families.  New  York:  Palgrave  Macmillan. 
Fullan,  M.  (2000).  Leadership  for  the  21st  century:  Breaking  the  bonds  of  dependency.  In  The  Jossey-Bass 
reader  on  educational  leadership  (pp.  156-164).  San  Francisco:  Jossey-Bass. 

Flyvbjerg,  B.  (2003).  Making  social  science  matter:  Why  social  science  fails  and  how  it  can  succeed  again. 
Cambridge:  England.  Cambridge  University  Press. 

Garcea,  J.  (1997,  June.).  Saskatchewan’s  aborted  municipal  service  act  (Bill  33):  Pegasus  or  Trojan  horse? 

Paper  presented  at  the  Canadian  Political  Science  Association  Annual  Meeting,  St.  John’s, 
Newfoundland. 

Hall,  I.  (1994).  Policy  networks  and  policy  communities.  Unpublished  Doctoral  Dissertation.  Saskatoon:  SK. 
University  of  Saskatchewan. 

Hargreaves,  D.  H.  (2001).  A capital  theory  of  school  effectiveness  and  improvement.  British  Educational 
Research  Journal,  27,  (4),  487-502. 

Howlett,  M.,  & Ramesh,  M(1995).  Studying  public  policy:  policy  cycles  and  policy  subsystems.  New  York: 
Oxford. 

Lane,  J.  E.  (1995).  New  public  management.  London,  England:  Routledge. 

Ibarra,  H.  (1992).  Structural  alignments,  individual  strategies,  and  managerial  action.  In  N.  Nohria  (Ed.), 
Networks  and  organizations  (pp.  165-188).  Boston,  MA:  Harvard  Business  School  Press. 

Kowch,  E.  (2003).  Policy  Networks  and  Policy  Communities  in  Three  Western  Canada  Universities:  Neo 


511 


Institutional  Responses  to  a Pan  Institutional  Issue.  University  of  Saskatchewan.  Doctoral 
Dissertation. 

Reigeluth,  C.  M.  (2001).  What  every  AECT  member  needs  to  know  about  systemic  change:  The  beginning  of  a 
dialog.  TechTrends,  46  (1),  12-15. 

Rhodes,  R.  A.  (1996).  From  institutions  to  dogma:  Tradition,  eclecticism  and  ideology  in  the  study  of  British 
public  administration.  Public  administration  review,  56,  507-523. 

Salomon,  D.  (2000).  Toward  a post-modern  agenda  in  instructional  technology.  Educational  Technology 
Research  and  Development,  48,  (4),  5-20. 

Senge,  P.  (2000).  Give  me  a lever  long  enough  and  single-handed  I can  move  the  world.  In  The  Jossey-Bass 
reader  on  educational  leadership  (pp.  13-26).  San  Francisco:  Jossey-Bass. 

Van  Wijk,  R.,  Van  Den  Bosch,  F.,  & Volberda,  H.  (2003).  Knowledge  and  networks.  In  Mark  Easterby-Smith 
and  Marjorie  Lyles  (Eds.)  Handbook  of  Organizational  Learning  and  Knowledge  Management  (pp. 
428-  454).  Oxford:  Blackwell  Publishing. 

Wheatley,  M.  (2000).  Good-bye,  command  and  control.  In  The  Jossey-Bass  Reader  on  Educational  Leadership 
(pp.,  339-348).  San  Francisco:  Jossey-Bass. 

Woolcock,  M.  & Narayan,  D.  (2000)  Social  capital:  implications  for  research  and  policy.  World  Bank  Research 
Observer,  15:  225-249. 


512 


Collaborative  Learning  Experiences  in  Online  Instructional  Design 

Courses 

Heng-Yu  Ku 
Linda  Lohr 
Yi-Chia  Cheng 

University  of  Northern  Colorado 

Abstract 

Few  universities  are  currently  offering  an  online  course  with  collaborative  learning  component  in 
Instructional  Design.  This  study  shared  information  on  an  on-going  evaluation  of  online  Instructional  Design 
courses  from  94  graduate  students  who  have  taken  this  course,  hoping  to  find  out  the  most  effective  way  to 
deliver  similar  courses.  This  research  focused  on  how  effectively  the  instructor  integrated  various  functions  of 
Blackboard  and  other  teaching  and  managing  strategies  into  the  online  course.  Findings  on  students’  attitudes 
toward  this  course  and  strategies  for  building  online  collaborative  learning  communities  from  both  the 
students’  and  instructor’s  points  of  view  were  discussed  and  explored. 

Introduction 

Effective  learning  is  not  simply  about  the  transfer  of  knowledge  but  about  developing  skills  for  life- 
long learning  (Vargo,  1997).  Although  learning  can  take  place  in  any  environment,  Ramsden  (1992)  suggested 
that  effective  education  is  based  on  deep  learning  that  is  learner-centered,  active,  and  in  context.  Web-based 
courses  claim  to  have  moved  learning  from  instructor-centered  to  learner-centered  approaches  and  require 
learners  to  be  self-disciplined  to  maximize  their  learning. 

Distance  education  has  developed  dramatically  during  the  past  few  years  through  the  application  of 
learning  theory  to  the  delivery  of  materials.  Baker  (1995)  indicated  that  interaction  is  important  for  a variety  of 
learning  types,  level  of  learning  satisfaction,  and  persistence.  Interaction  is  central  to  the  expectations  of 
teachers  and  learners  in  distance  education  and  is  a primary  goal  of  the  educational  process  (Berge,  2002). 
However,  students  perceived  too  much  interaction  as  frustrating  busy  work,  whereas  too  little  interaction  might 
cause  isolation  (Berge,  1999). 

According  to  Moore  (2003),  there  are  four  types  of  interaction  identified  in  the  literature:  learner- 
content  interaction,  learner-instructor  interaction,  earner-learner  interaction,  and  learner- interface  interaction. 
This  research  study  investigated  all  four  types  of  interaction  in  online  courses  but  specifically  focused  on  the 
third  type  of  interaction:  Learner-learner  interaction.  In  a study  conducted  by  Northrup  (2002),  it  was  found  that 
participants  liked  to  discuss  ideas  and  concepts  as  well  as  to  share  information  with  their  peers.  Participants 
considered  promoting  online  collaboration  and  conversation  an  important  attribute  of  distance  learning. 

Traditional  pedagogical  approaches  in  education  have  decontextualized  knowledge  and  skills  to  real- 
world  application  (McLoughlin,  2002).  Candy,  Crebert,  and  O’Leary  (1994)  underscored  that  university 
education  should  develop  a capacity  for  and  understanding  of  teamwork  along  with  critical  thinking.  This 
situation  calls  for  educators  to  develop  activities  that  support  group  collaboration  (Bennett,  2004).  Working 
collaboratively  helps  students  to  take  into  consideration  different  perspectives  while  building  a deep 
understanding.  It  also  reflects  how  people  work  in  real-world  contexts  and  how  practitioners  share  knowledge 
within  a community  (Brown,  Collins,  & Duguid,  1989;  Duffy  & Cunningham,  1996;  Lave  & Wenger,  1991). 

Collaborative  learning  refers  to  an  instructional  method  in  which  small  groups  of  learners  mutually 
engage  in  the  learning  environment  to  accomplish  a shared  goal  (Abrami  & Bures,  1996;  Bruffee,  1993; 

Murphy,  Cifuentes,  & Shih,  2001,  Tu  & Corry,  2002).  Collaboration  should  at  least  contain  sharing  the  learning 
tasks,  combining  expertise,  knowledge,  and  skills,  and  building  a learning  community  (Bernard,  Rojo  de 
Rubalcava,  & St-Pierre,  2000;  Slavin,  1995).  The  advantages  of  collaborative  learning  include  the 
encouragement  of  active  and  constructive  learning,  deep  processing  of  information,  critical  thinking,  and  goal- 
based  learning  (Bernard  et  al.,  2000).  Slavin  (1995)  asserted  that  collaborative  learning  enhances  the 
opportunity  to  combine  expertise,  share  knowledge  and  skills,  and  build  a learning  community.  Despite  these 
benefits  and  the  massive  literature  base  advocating  collaborative  learning,  researchers  have  pointed  out  that 


513 


collaborative  groups  frequently  do  not  work  well  (Salomon,  1992). 

Online  collaboration  can  be  defined  as  the  collaborative  learning  that  takes  place  in  a distance-learning 
environment.  A critical  factor  to  the  success  of  online  collaboration  is  the  feeling  of  learners  being  engaged  in  a 
leaning  community  (Yang,  2002).  Hasler-Waters  and  Napier  (2002)  contended  that  receiving  support,  getting 
acquainted,  establishing  communication,  building  trust,  and  getting  organized  are  elements  that  foster 
successful  online  teams.  Although  online  group  collaboration  can  generate  new  knowledge,  attitudes,  and 
behaviors,  it  requires  significantly  more  time  and  effort  than  traditional  learning  (Kulp,  1999). 

Previous  studies  revealed  both  positive  and  negative  student  perspectives  toward  online  collaborative 
learning.  Students  expressed  that  their  communication  skills  and  problem-solving  skills  were  improved  through 
online  collaboration  (Yang,  2002).  In  Kitchen  and  McDougall’s  study  (1999),  students  reported  enjoying  the 
convenience  and  opportunity  to  collaborate  online.  However,  some  described  their  dissatisfaction  regarding  the 
instructional  strategies  and  the  delivery  methods.  Research  also  disclosed  that  learners  tend  to  resist  group 
collaboration  because  the  outcomes  depend  on  the  input  of  other  group  members  (Ko  & Rossen,  2001). 

Instructional  Design  is  a compulsory  course  for  graduate  students  in  the  field  of  Educational 
Technology,  unanimously  regarded  as  one  of  the  most  difficult  courses.  Students  typically  learn  how  to  design 
an  instructional  lesson  or  module.  From  creating  initial  design  documents  to  the  ultimate  actual  lessons,  it  is  not 
uncommon  to  hear  students  complaining  how  confusing  the  whole  process  is.  In  fall  2001,  a university  in  the 
mid- western  area  of  the  United  States  offered  this  course  online  for  the  first  time,  using  a delivery  platform 
called  Blackboard.  The  Blackboard  program  used  to  deliver  online  Instructional  Design  courses  provided  web- 
based  tools  that  made  communication  and  other  collaborative  exercises  easier  for  online  teachers  and  students. 

There  has  been  limited  research  that  reports  student  perceptions  and  attitudes  toward  their  online 
collaboration  experiences  as  well  as  what  factors  students  consider  crucial  in  an  online  learning  environment. 
What  exists  has  mostly  focused  on  student  perspectives  with  their  online  learning  experiences.  This  research 
intended  to  examine  online  collaborating  learning  experiences  from  students  ’ point  of  view  to  provide  a basis 
for  evaluating  the  effectiveness  of  online  instructional  design  courses  and  to  provide  suggestions  and  strategies 
that  instructors  could  implement  in  their  online  courses.  The  following  research  questions  were  addressed: 

1.  What  were  student  perceptions  and  attitudes  toward  taking  an  online  course  in  instructional 
design? 

2.  What  were  student  attitudes  toward  working  in  a collaborative  setting  in  the  online  environment? 

3.  What  elements  did  students  consider  critical  for  a successful  online  course? 

Method 

Subjects 

The  subjects  were  94  mid  -western  graduate  students  enrolled  in  an  online  course  in  Instructional 
Design.  Data  were  collected  between  the  years  of  2002  to  2004.  Eighty  percent  of  the  subjects  were  either 
majoring  in  Educational  Technology  or  Educational  media.  Sixty  four  females  and  30  males  participated. 

Eighty  of  the  students  were  American  and  14  were  international.  Less  than  five  percent  of  these  students  had 
experience  with  taking  completely  online  courses. 

Online  Course  Format 

The  instructor  delivered  this  course  using  a web-based  course  management  system  called  Blackboard. 
The  interface  of  Blackboard  is  shown  in  Figure  1.  For  important  announcements,  the  instructor  would  post  each 
announcement  under  the  “Announcements”  function  as  well  as  email  the  same  announcement  to  each  student  in 
case  students  failed  to  login  in  Blackboard  that  day.  To  remind  students  of  course  objectives,  activities,  and 
requirements,  a course  syllabus  was  posted  under  the  “Syllabus”  function.  To  create  online  communities  among 
students,  the  instructor  asked  students  to  email  their  biographies  and  pictures  to  the  instructor  before  the  end  of 
the  first  week.  The  instructor  posted  each  student’s  biography,  picture,  and  contact  information  under  “Faculty 
Information”  function  and  encourage  students  to  view  other  students’  information  on  Blackboard  and  contact 
each  other. 

To  keep  students  on  task,  the  instructor  used  the  “Assignments”  function  to  post  information  and 
inform  student  what  weekly  activities  and  readings  should  be  completed.  To  encourage  interaction  and  build  an 
online  community,  the  “Communication”,  “Chat”  and  “Discussion  Board”  functions  offer  commonplaces  for 
the  instructor  and  students  to  post  questions  and  share  ideas  with  each  other. 

To  offer  information  other  than  readings  from  a required  textbook,  the  instructor  also  developed 
weekly  mini-lectures  that  synthesized  important  textbook  information.  These  mini-lessons  as  well  as  examples 
of  design  documents  and  self-paced  lessons  were  posted  in  a “Course  Material”  function.  For  additional 
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information  that  related  to  a specific  topic,  the  instructor  would  post  supporting  web  links  under  the  “Web 
Sites”  function.  The  instructor  also  posted  the  grade  for  each  assignment  under  the  “Student  Tool”  function  to 
allow  students  to  check  grades  online. 

To  encourage  collaboration  and  increase  interaction  among  students  in  this  online  course,  the  students 
were  asked  to  form  groups  of  three  and  to  send  the  names  of  their  group  members  to  the  instructor  by  the  end  of 
the  second  week.  For  students  who  did  not  send  names,  the  instructor  would  randomly  assign  three  people  to 
form  a group.  Each  group  would  then  decide  on  a topic  of  interest  and  create  a design  document  and  self-paced 
lesson  for  that  particular  topic  throughout  the  semester.  The  instructor  used  the  “Groups”  function  and  placed 
those  students  together  as  one  group.  From  there,  group  members  had  access  to  participating  in  a synchronous 
group  chat  room,  posting  messages  under  the  group  discussion  board,  sending  email  to  selected  group  members 
or  the  whole  group,  and  posting  assignments  via  file  exchange.  In  order  to  encourage  equal  contribution  among 
students,  all  students  were  informed  in  the  beginning  of  the  semester  that  instructor,  self,  and  peer  evaluation 
would  be  counted  as  20  percent  of  their  final  grade. 

In  the  process  of  creating  a design  document  of  the  chosen  topic,  each  group  was  required  to  work  on 
the  draft  design  document  for  three  assignments.  The  first  assignment  covered  needs  assessment,  learner 
analysis,  contextual  analysis,  and  task  analysis.  The  second  assignment  contained  instructional  objectives, 
questions  and  feedback.  The  third  assignment  included  instructional  sequencing,  instructional  strategies,  and 
message  design.  Each  group  would  provide  feedback  to  and  receive  feedback  from  their  group  members,  revise 
their  first  drafts  based  on  the  peer  feedback,  and  post  their  revised  drafts  online  via  file  exchange  under  the 
“Groups”  function.  Posting  assignments  on  the  file  exchange  allowed  the  group  members  and  the  instructor  to 
access  documents  for  reading.  Following  the  posting  of  these  drafts,  the  instructor  would  look  over  the  revised 
draft  of  the  assignment  that  each  group  produced  and  provide  feedback  to  each  group.  Students  would  then 
modify  drafts  based  on  the  instructor’s  feedback. 

The  same  procedures  were  repeated  for  each  assignment  and  students  would  compile  all  revised 
assignments  together  into  a final  design  document.  After  all  sections  of  the  design  process  were  covered, 
students  would  develop  a self-paced  lesson  based  on  the  design  document  that  they  had  been  developing. 
Students  would  then  conduct  a formative  evaluation  to  test  the  draft  of  the  self-paced  lesson  to  its  target 
audience  and  write  up  an  evaluation  report.  Students  would  then  use  the  evaluation  results  and  learner  feedback 
to  revise  their  self-paced  lessons  and  design  documents.  Finally,  students  submitted  the  final  version  of  the 
design  document  and  self-paced  lesson  during  the  last  week  of  the  semester. 

Materials 

During  the  last  week  of  each  semester,  students  completed  a 20 -item  Student  Attitude  Survey  designed 
for  this  study  to  indicate  their  attitudes  toward  the  online  learning  environment  and  their  general  attitudes 
toward  this  course.  These  items  were  5-point  Likert -scale  items  that  ranged  from  1 (strongly  disagree)  to  5 
(strongly  agree).  The  KR-20  reliability  coefficient  for  the  20  Likert -type  items  was  .87. 

The  second  part  of  the  survey  was  comprised  of  five  open-ended  questions  dealing  with  student 
perceptions  toward  online  learning  environment,  online  collaborated  setting,  working  on  group  projects,  and 
suggestions  on  the  important  elements  that  a successful  online  course  should  comprise.  These  questions  were:  1. 
What  did  you  like  most  about  the  online  setting?  2.  What  did  you  like  least  about  the  online  setting?  3.  Did  you 
like  or  dislike  learning  in  an  online  collaborative  setting?  Why  or  why  not?  4.  Do  you  think  you  would  have 
learned  more  in  this  class  if  you  had  done  your  project  alone?,  and  5.  You  have  just  lived  through  a fully  online 
course.  In  your  opinion,  what  do  you  consider  as  a successful  online  course?  What  elements  should  be  there? 

Procedure 

Data  was  collected  from  the  Student  Attitude  Survey  across  five  semesters  of  a graduate  level 
instructional  design  course.  In  these  full-semester  courses  students  worked  in  small  groups  to  collaboratively 
create  instructional  units.  The  process  of  teaching  this  online  course  has  been  observed  and  recorded  in  detail  by 
the  first  author.  The  Student  Attitude  Survey  was  sent  out  as  an  email  attachment  to  students  during  the  final 
week  of  the  each  semester.  All  participants  filled  out  the  20 -item  Student  Attitude  Survey  and  responded  to  five 
open-ended  questions  and  sent  their  responses  as  an  email  attachment  to  their  instructor  by  the  last  day  of  the 
semester. 

Data  Analysis 

From  the  Student  Attitude  Survey,  student  responses  were  calculated  and  ranked  for  each  survey  item.  From  the 
five  open-ended  questions,  a thematic  analysis  was  conducted  to  identify  emerging  themes  and  patterns  for 
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responses  of  each  question.  Furthermore,  the  recurring  responses  were  categorized  and  counted  and  provide  the 
framework  for  discussion. 


Results 


Student  Attitude  Survey 

The  means  and  standard  deviations  for  the  20-item  Student  Attitude  Survey  were  calculated  and 
reported  in  Table  1.  The  overall  mean  score  across  the  Student  Attitude  Survey  items  was  3.86,  a rating 
indicating  agreement  with  positive  statements  about  this  course.  The  five  highest-rated  statements  on  the  survey 
were  “I  like  the  mini-lectures  provided  by  the  instructors”  (M  = 4.43,  SD  = .65),  “I  liked  the  File  Exchange 
function  on  Blackboard”  (M  = 4.42,  SD  = .66),  “I  like  to  see  the  short  biography  of  my  instructors  and 
classmates  on  Blackboard”  (M  = 4.40,  SD  = .66),  “I  liked  the  Announcement  function  on  Blackboard  (M  = 

4.39,  SD  = .66),  and  ‘‘I  liked  the  feedback  that  my  instructors  provided  (M  = 4.32,  SD  = .69).”  The  five  lowest- 
rated  statements  were  “This  course  was  easy”  (M  = 2.20,  SD  = .91),  “ I liked  the  textbook  that  we  used  in  this 
course”  (M  = 3.41,  SD  = .92),  “I  liked  the  group  format  in  this  course”  (M  = 3.45,  SD  = 1.31),  “I  liked  the 
online  environment  of  the  course”  (M  = 3.46,  SD  = 1.19),  and  “I  would  take  this  course  as  an  online  course 
again”  (M  = 3.52,  SD  = 1.26). 


Table  1 Student  Attitude  Survey  Scores 


Statement 

M 

SD 

1.  I liked  the  mini-lectures  provided  by  the  instructors. 

4.43 

.65 

2.  I liked  the  File  Exchange  function  on  Blackboard. 

4.42 

.66 

3.  I like  to  see  the  short  biography  of  my  instructors  and 

4.40 

.66 

classmates  on  Blackboard. 

4.  I liked  the  Announcement  function  on  Blackboard. 

4.39 

.66 

5.  I liked  the  feedback  that  my  instructors  provided. 

4.32 

.69 

6.  I like  to  see  pictures  of  my  instructors  and  classmates  on 

4.27 

.72 

Blackboard. 

7.  I learned  a lot  from  this  course. 

4.11 

.77 

8.  I like  to  receive  feedback  from  my  group  members. 

4.09 

.86 

9.  I would  like  to  meet  with  my  instructors  and  classmates  face-to- 

4.06 

.89 

face  some  day. 

10.  The  grading  was  fair  in  this  course. 

3.99 

.80 

11.  I liked  this  course. 

3.97 

.89 

12.  I spent  more  time  working  on  this  course  than  my  other 

3.76 

1.02 

courses. 

13.  I like  to  provide  feedback  to  my  group  members. 

3.68 

1.05 

14.  The  amount  of  the  work  required  was  fair. 

3.65 

.90 

15.  I would  recommend  this  online  course  to  others. 

3.53 

1.13 

16.  I would  take  this  course  as  an  online  course  again. 

3.52 

1.26 

17.  I liked  the  online  environment  of  the  course. 

3.46 

1.19 
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1.31 


18.  I liked  the  group  format  in  this  course.  3.45 

19.  I liked  the  textbook  that  we  used  in  this  course.  3.41  .92 

20.  This  course  was  easy.  2.20  .91 

Total  186  A9 

Note.  Reponses  ranged  from  1 (Strongly  Disagree)  to  5 (Strongly  Agree). 


Attitudes  Toward  Online  Course  (Likes  and  Dislikes) 

When  asked  what  students  liked  most  about  this  online  course,  flexibility,  convenience,  easy 
communication,  semi -constructive  nature  of  the  course,  group  member  and  instructor  feedback,  weekly  mini- 
lectures, project  examples,  face-to-face  meetings  with  group  members,  and  studying  at  their  own  pace  were  the 
what  they  liked  the  most  about  this  course.  They  also  liked  the  fact  that  instructors  posted  each  student’s  picture 
and  biography  on  Blackboard  in  order  for  them  to  know  each  other  and  to  cultivate  an  online  community. 

When  asked  what  students  liked  least  about  this  course,  some  of  them  indicated  lack  of  immediate 
interaction  and  feedback,  isolation  during  the  learning  process,  the  fact  that  some  group  members  failed  to 
provide  constructive  feedback  on  time,  coordinating  with  group  members,  technical  difficulties,  the  textbook, 
and  inadequate  computer  knowledge  diminished  their  enjoyment  of  this  course.  They  also  expressed  that  they 
missed  the  active  class  atmosphere  where  they  were  able  to  raise  questions  in  class  to  discuss  with  classmates 
and  instructors . 

Attitudes  Toward  Online  Collaborated  Setting  (Likes  and  Dislikes) 

When  students  were  asked  whether  they  liked  or  disliked  learning  in  an  online  collaborated  setting,  32 
students  (34%)  liked  learning  in  an  online  collaborated  setting,  47  students  (47%)  disliked  learning  in  an  online 
collaborated  setting,  and  14  students  (16%)  had  mixed  feelings. 

Students  who  liked  learning  in  an  online  collaborative  setting  appreciated  having  group  members  that 
they  could  bounce  ideas  with  and  having  opportunities  to  provide  and  receive  feedback  from  others.  In  that 
way,  they  felt  that  they  were  “forced”  and  had  responsibilities  to  read  the  chapters  and  course  materials 
thoroughly  so  they  could  provide  constructive  feedback  to  their  group  members.  Some  positive  comments  from 
students  regarding  learning  in  an  online  collaborated  setting  were: 

I really  enjoyed  working  with  partners  as  we  bounced  ideas  and  feedback  off  of  each  other  to  create,  what  I feel, 
is  a quality  project.  We  worked  we  11  together  and  came  up  with  ideas  we  could  not  have  if  we  were  working 
independently.  Our  willingness  to  work  together,  combining  our  resources,  greatly  helped  our  overall  product. 

The  collaborative  piece  of  this  of  this  course  was  critical  to  avoid  total  frustration  and  annoyance. 
Initially,  I found  the  information  and  language  very  confusing.  With  a group  to  bounce  off  of,  it  didn’t  feel  quite 
so  hopeless.  The  exchange  of  ideas  led  us  to  a much  better  product  than  any  of  us  would  have  created  alone. 
Also,  this  approach  mimics  team  teaching  which  is  the  environment  most  of  us  inhabit  in  our  schools  making  it 
more  authentic. 

For  those  students  who  disliked  learning  in  an  online  collaborated  setting,  some  of  their  reasons  were 
the  ineffective  and  inefficient  communication,  uneven  workloads  and  efforts,  difficulty  adjusting  to  each  other’s 
schedules,  the  time  consuming  nature  of  the  class,  and  arguing  with  group  members  on  ideas.  Some  negative 
comments  from  students  regarding  learning  in  an  online  collaborated  setting  were: 

I normally  work  very  well  in  groups  and  enjoy  the  group  setting.  However,  it  is  very  hard  to  be  in  a 
group  with  complete  strangers  just  over  the  Internet.  I felt  at  times  that  I was  doing  most  of  the  work 
and  they  weren’t  putting  as  much  effort  into  the  project  as  I was....  Communication  was  also  difficult 
because  I couldn’t  explain  my  ideas  in  the  way  that  I would  have  been  if  we  were  to  meet  face  to  face. 
The  whole  process  was  very  frustrating! 

I liked  the  online  setting,  but  disliked  the  collaboration.  If  you  are  teamed  with  people  who  are  shooting  for  a 
"B'  or  just  to  pass  the  class,  it  is  difficult  to  get  an  ‘A’  for  a group  assignment  without  taking  on  the  majority  of 
the  work.  I feel  that  I shouldered  the  vast  majority  of  the  work  for  the  group.  In  this  way,  this  online  class  was 
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far  more  work  than  a traditional  class.  I do  not  know  how  I would  feel  if  it  was  an  online,  non-collaborative 
class.  It  is  hard  to  distribute  the  work  evenly  in  an  online  class... 

Some  comments  from  students  with  mixed  feelings  regarding  learning  in  an  online  collaborated  setting 

were: 


I dislike  having  to  rely  on  others  (and  their  schedules)  to  complete  assignments,  especially  when  it 
impacted  my  schedule  and  my  grades.  On  the  other  hand,  I do  like  getting  feedback  from  others  and 
being  able  to  work  at  my  own  pace  at  times  that  are  convenient  for  me. 

I must  admit,  at  first  I absolutely  hated  it.  (I  know  that’s  a strong  word,  but  I did.)  If  the  course  had  not 
been  a requirement,  I would  have  dropped  it.  Communication  felt  exasperating.  But  with  time,  and 
getting  to  know  my  teammates,  it  got  easier.  We  worked  out  the  kinks.  I had  great  teammates.  (I  don’t 
believe  that  is  always  the  case.)  When  one  of  us  was  stuck,  one  always  came  through.  (I  can’t  imagine 
what  this  would  be  like  if  there  was  someone  not  doing  their  part!)  Now,  I miss  them.  I wrote  them 
yesterday  to  tell  them  I thought  I was  having  ID  withdrawals! 

Attitudes  Toward  Online  Collaborated  Learning 

When  asked  whether  students  would  have  learned  more  in  this  class  if  they  had  done  their  project 
alone,  70  students  (75%)  said  “No”,  14  students  (15%)  indicated  “Yes”,  and  10  students  (11%)  kept  their 
opinion  as  neutral.  From  student  responses,  we  also  discovered  that  in  order  for  students  working  well  in  groups 
in  an  online  collaborative  setting,  the  five  Cs  (Communicate,  Cooperate,  Compromise,  Compliment,  and 
Commitment)  need  to  be  included. 

The  first  C is  to  Communicate  and  students  mentioned,  “We  instantly  established  a routine  that  was 
very  focused  on  the  task  at  hand;  we  were  able  to  be  honest  in  working  with  each  other  and  truly  developed  a 
cordial,  often  fun  working  relationship.”  and  “...by  having  to  work  with  others,  I had  to  exercise  people  skills 
and  learn  to  get  alone  and  say  things  in  persuasive  rather  than  confrontive  ways...” 

The  second  C is  to  Cooperate  and  students  expressed,  “It’s  always  great  to  have  someone  else  be  the 
sounding  board,  especially  when  they  have  just  as  much  ownership  in  the  assignment.”,  “Having  group  ideas 
and  a checks  and  balance  system  really  worked  well.  I learned  more  hashing  out  the  details  with  my  group  than 
I would  have  on  my  own.”,  and  “I  really  feel  our  final  product  was  better  for  the  added  insights  and  creativity  of 
three  minds  instead  of  one ....  I think  each  of  us  benefited  from  the  camaraderie  we  experienced,  and  the 
support.” 

The  third  C is  to  Compromise  and  students  stated,  “It  meant  compromise,  especially  in  the  area  of 
topic  selection  as  we  all  had  our  own  content  we  wanted  to  deal  with. . . ” and  “I  think  being  forced  together  in  a 
group,  not  of  our  own  choosing,  best  simulated  the  business  environment.  We  were  forced  to  cooperate, 
compromise  and  communicate  with  each  other  in  a way  that  working  alone  would  not  allow.” 

The  fourth  C is  to  Compliment  each  other  and  students  indicated,  “We  all  really  complimented  each 
other.  I must  admit  I was  stuck  two  times  while  doing  this  project.  One  of  the  other  teammates  got  the  ball 
rolling  and  I think  they  would  say  the  same  thing  about  me  at  times  when  they  were  stuck  “,  “We  had  strengths 
that  complimented  each  other,  so  we  got  to  see  the  whole  picture  and  fill  in  the  gaps”,  and  “...when  working  in 
a group  you  can  draw  on  the  strengths  of  the  individual  group  members.  In  our  group  one  member  was  a better 
writer,  one  had  more  experience  with  power  point,  etc.  Plus  we  could  all  draw  from  personal  experiences.” 

The  fifth  C is  to  Commit  to  the  team  and  student  commented,  “...having  members  that  work  as  hard  as 
you  and  are  as  committed  makes  all  the  difference”  and  “Having  the  advantage  of  each  teammate  contributing 
his/her  different  perspectives  for  the  project  was  terrific.  Moreover,  we  supported  each  other  both  academically 
and  emotionally,  since  taking  an  on-line  course  was  very  challenging  to  us  novices.” 

Critical  Elements  in  an  Online  Course 

When  asked  about  what  students  considered  as  critical  elements  in  a successful  online  course,  their  top 
ten  comments  included:  1.  Frequent  instructor-to-student  and  peer-to-peer  communication  (55%),  2.  Clear 
objectives,  materials,  and  course  outlines  (33%),  3.  Useful  mini-lectures  (20%),  4.  Strong  instructor  support 
(18%),  5.  Opportunities  to  access  and  view  previous  project  examples  (18%),  6.  User-friendly  features  on 
Blackboard  (12%),  7.  Superior  organizational  skills  (12%),  8.  Just  in  time  resources  (11%),  9.  Proficiencies  in 
technology  (10%),  and  10.  Periodic  online  discussion  (10%).  In  addition,  posting  pictures  and  bios  of  students 
and  faculty,  clarifying  project  deadlines,  mastering  better  self-regulation  and  self-efficacy  traits,  and  having  the 
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opportunity  to  see  other  groups’  projects  (>  7%)  were  important  attributes  that  student  considered  as  crucial 
elements  in  a successful  online  course. 


Discussion 

This  research  focused  on  how  effectively  the  instructor  integrated  various  functions  of  Blackboard  and 
other  teaching  and  managing  strategies  into  an  online  course  in  Instructional  Design.  Findings  on  students’ 
attitudes  toward  this  course  and  strategies  for  building  online  collaborative  learning  communities  from  both  the 
students’  and  instructor’s  points  of  view  were  discussed.  Although  many  of  the  findings  are  similar  to  previous 
studies,  less  frequently  observed  in  the  literature  are  the  actual  comments  that  indicate  a dislike  of  group  activity 
while  admitting  the  importance  of  it. 

According  to  Sirnoff  and  Maher  (1997),  a successful  online  learning  course  depends  on  (1)  delivering 
course  materials  to  students  in  time  and  (2)  providing  effective  communication  between  students  and 
instructors.  The  instructor  in  this  course  concurred  with  this  statement  and  posted  timely  mini-lectures  and 
project  examples.  The  instructor  also  incorporated  online  group  activities  to  encourage  communication  between 
students  (peer  feedback)  and  instructors  (instructor  feedback)  via  announcement,  email,  discussion  board,  file 
exchange,  and  chat  functions.  Interestingly,  the  students  also  ranked  the  “ I like  the  mini-lectures  provided  by 
the  instructors”  and  “I  liked  the  File  Exchange  function  on  Blackboard”  as  the  two  highest-rated  items  on  the 
Student  Attitude  Survey. 

Similar  to  other  research  findings  (Hiltz,  1998,  Howland  & Moore,  1998;  Yang  2002),  our  results 
indicate  that  convenience,  flexibility,  and  easy  communication  were  common  themes  in  the  positive  student 
responses  regarding  the  online  setting  while  communication  difficulties,  lack  of  face-to-face  interaction,  and 
sense  of  isolation  were  the  overriding  negative  themes  regarding  the  online  setting. 

When  asked  whether  students  liked  or  disliked  learning  in  an  online  collaborative  setting,  different 
opinions  were  noted.  Half  of  students  (50%)  indicated  they  disliked  learning  in  an  online  collaborative  setting 
while  one  third  of  students  (34%)  held  opposite  opinions.  The  finding  that  50  percent  of  students  disliked  the 
collaborative  setting  corresponds  with  the  statement  of  “I  liked  the  group  format  in  this  course”  on  the  Student 
Attitude  Survey  (M  = 3.45)  that  was  rated  as  the  third  lowest  items  on  the  Student  Attitude  Survey.  Contrarily, 
when  asked  whether  students  would  have  learned  more  in  this  class  if  they  had  done  their  project  individually, 
three  out  of  four  students  (75%)  felt  that  the  collaborative  environment  produced  greater  learning.  Such  findings 
emphasize  the  usefulness  and  importance  of  online  collaborative  learning. 

From  student  reactions  to  the  fourth  open-ended  question,  we  find  that  the  five  Cs:  Communicate, 
Cooperate,  Compromise,  Compliment,  and  Commitment  need  to  be  incorporated  within  the  group  setting  so 
group  members  can  have  better  working  relationship  with  each  other  to  produce  quality  projects  and  greater 
learning  in  an  online  collaborative  environment.  From  their  responses  to  the  last  open-ended  question,  we  also 
identified  the  top  ten  critical  elements  that  students  considered  in  a successful  online  course.  Overall,  students 
concurred  that  a solid  course  structure  (the  ten  critical  elements),  as  well  as  encouraging  and  supporting 
collaborative  project  development  (the  5 Cs),  leads  to  effective  learning  and  better  quality  of  the  final  project. 
We  have  provided  a model  for  online  collaborative  learning  plans  as  shown  in  Figure  1. 

Distance  learning  is  gaining  in  popularity  because  of  the  convenience  it  brings  and  many  academic 
institutions  place  more  and  more  emphasis  on  developing  online  learning.  However,  the  preparation  for 
teaching  online  classes  takes  time,  detailed  thought,  lots  of  patience,  and  adequate  computer  and 
communication  skills.  When  designing  the  online  teaching  materials,  instructors  have  to  take  interaction  and 
collaboration  into  consideration  and  encourage  interaction  and  support  communal  scaffolding  throughout  the 
collaboration  process.  Hopefully,  such  acts  will  motivate  students  in  the  online  collaboration  process  and  will 
make  the  collaboration  a worthwhile  learning  experience  for  them. 

The  results  of  the  study  have  practical  significance  for  helping  the  department  in  which  this  study  was 
conducted.  Guidelines  are  offered  for  instructors  planning  to  implement  online  collaborative  learning 
components  as  well  as  students  required  to  work  collaboratively  in  the  online  environment.  Furthermore,  it  may 
help  the  instructor  to  have  a more  systematic  understanding  of  the  pedagogical,  technological,  and 
administrative  approaches  to  distance  learning.  Future  research  can  explore  various  online  teaching  strategies  to 
help  student  work  well  collaboratively  and  produce  better  outcomes  in  an  online  learning  environment. 
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Figure  1.  Online  collaborative  learning  model. 
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Exploring  the  Potential  of  WAP  Technology  in  Online  Discussion 

Chwee  Beng  Lee 
University  of  Missouri- Columbia 

Introduction 

The  intent  of  this  study  is  to  explore  how  WAP  (wireless  application  protocol)  technology  mediates 
online  discussions.  The  key  focuses  of  this  research  are  on  the  implications  of  WAP  technology  for  online 
discussions,  the  types  of  discussion  topics  that  are  most  suitable  for  WAP-based  discussions  and  the  finding  of 
the  combination  of  WAP-  and  WEB-based  discussions. 

Multiple  methods  of  inquiry  were  employed  in  this  study.  A survey,  face-to-face  and  focus  group 
interviews  were  conducted  to  find  out  the  participants’  perceptions  of  using  the  WAP-enabled  mobile  phone  as 
a communication  tool,  and  how  such  a tool  has  facilitated  or  impeded  their  learning  processes  both  technically 
and  cognitively.  A content  analysis  was  made  on  the  postings  generated  from  WAP-  and  WEB-based  forums. 

This  study  is  one  of  the  very  first  attempts  at  investigating  the  use  of  WAP-enabled  mobile  phones  for 
online  discussions  in  one  of  the  higher  education  institutions  in  Singapore.  By  exploring  the  potential  of  this 
newly  developed  technology,  it  is  hoped  that  this  study  may  provide  guidelines  to  educators  as  they  reflect  on 
the  way  online  discussions  can  be  integrated  into  a course  and  also  laying  a foundation  for  future  development 
of  a mobile  online  learning  environment. 

Online  Discussion  as  a Sociocultural  Tool 

In  this  study,  participants  were  requested  to  participate  in  asynchronous  online  discussions. 
Asynchronous  communication  refers  to  anytime  -anywhere  communication  between  two  or  more  individuals.  In 
such  a communication,  participants  are  unrestrained  by  space,  time  and  pace.  They  may  read  the  messages  that 
are  posted  in  a central  location  or  delivered  to  their  email  box  at  their  own  convenience.  Wireless  technologies 
can  free  learners  from  the  need  to  be  tied  to  a particular  hard-wired  location  to  access  information 
(Gunawardena  & Msisaac,  2004).  There  are  several  common  forms  of  asynchronous  communications.  These 
include  e-mailing,  threaded  discussion  and  newsgroup.  In  this  paper,  online  discussions  refer  to  the  use  of  a 
web-based  application  that  enables  participants  to  create  and  edit  messages  that  are  stored  in  an  area  that  is 
accessible  to  group  members  who  organize  messages  into  “threads”  of  conversation  (Curtis  & Lawson,  2001). 

In  this  study,  the  participants  were  given  the  option  to  participate  either  in  the  WAP-  or  WEB-based  forums  via 
WAP-enabled  mobile  phones  or  their  own  personal  computers. 

Asynchronous  online  discussion  offers  a range  of  advantages  to  learners  such  as  self-paced 
participation  and  reflective  thinking.  In  an  online  discussion  environment,  students  are  likely  to  obtain  more 
experiences  managing  their  interactions  with  the  content  thems  elves.  Online  discussions  require  learners  to 
manage  their  own  learning,  free  from  the  teacher-centered  settings;  this  then  provides  an  opportunity  for 
learners  to  progress  independently  (Lee,  2002).  Researchers  also  found  that  online  discussions  are  more  task- 
oriented  than  face-to-face  discussions.  Also,  reading  and  writing  are  employed  discursively  as  a means  of 
focusing  members  of  a classroom  community  on  matters  of  joint  interest  (Lapadat,  2000).  Moreover,  as 
asynchronous  discussion  is  text -based,  the  meaning  of  a text -only  message  is  divorced  from  the  sender’s 
physical  presence  and  verbal  delivery  style  and  all  that  remains  is  what  the  person  actually  says  (Altanus,  1997). 

This  study  does  not  intend  to  make  comparison  between  different  technologies  on  how  they  perform  in 
online  discussions.  Rather,  it  seeks  to  explore  how  the  various  technologies  compliment  each  other  and  mediate 
online  discussions.  When  engage  in  online  discussions,  learners  should  be  given  more  avenues  to  log  on  to 
participate.  Web-based  course  when  combined  with  other  CMC  tools  such  as  email  or  bulletin  board  allows  the 
learners  to  learn  and  follow  their  own  path,  enriching  the  exchange  of  ideas  among  learners  (Box  ,1999). 

One  of  the  key  concepts  in  Vygotsky’s  work  is  the  zone  of  proximal  development  (ZPD).  The  ZPD  is 
the  range  of  difference  between  what  an  individual  may  accomplish  in  an  activity  or  task  alone  and  what  he  or 
she  may  accomplish  in  the  company  of  others  (Althauser  & Matuga,  1998).  It  is  in  the  ZPD  that  scaffolded 
learning  takes  place  to  support  learning.  Scaffold  assistance  can  come  from  both  cultural  tools  and  more 
knowledgeable  peers  or  experts  in  one’s  learning  environment  (Jarvela  & Hakkinen,  2000). 

Sociocultural  researchers  point  out  that  instruction  should  take  place  in  an  environment  in  which 
learners  use  socially  mediated  and  intellectual  tools  to  achieve  cognitive  development  (Rogoff  1990;  Salomon 
1993).  Bonk  and  Cunningham  ( 1998)  comments  that  collaborative  technologies  can  offer  opportunities  for  both 
peer  and  mentor  electronic  guidance  and  feedback  that  stimulate  student  discussions  and  internal  reflections  in  a 
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scaffold  learning  activity.  It  is  through  online  discussion  that  they  may  “voice  their  opinions  and  reflect  on  their 
learning,  thereby  increasing  inter-psychological  and  intra -psychological  activities  to  promote  individual’s 
cognitive  development”  (Zhu  1998,  p.234).  Online  discussion  is  a key  mediational  tool  for  “external  display  of 
students  thinking  processes  and  interchanges”  (Jarvela  and  Hakkinen  2000,  p.8).  It  promotes  “exciting  online 
learning  communities”  (King  1998,  p.368). 

Discussions  on  brainstorming  and  case  studies  were  adopted  in  this  study  based  on  the  rationale  that 
both  types  of  discussion  could  enrich  the  online  discussions  by  offering  students  opportunities  to  engage  in 
higher  levels  of  thinking  such  as  critical  thinking.  Chong  (1998)  reported  in  her  study  that  participants  claimed 
that  they  have  learned  tremendously  from  the  magazine  articles  and  that  authentic  cases  taken  from  real-life 
provided  the  complexity  necessary  to  encourage  critical  thinking  and  logical  argumentation.  Authenticity  in 
case  studies  did  provide  a challenging  and  real  life  learning  environment.  Also,  empirical  investigations  have 
found  that  electronic  brainstorming  groups  have  generated  more  ideas  than  verbally  brainstorming  groups, 
particularly  for  larger  group  sizes  (Gallupe,  Bastianutti  & Cooper,  1991;  Gallupe,  Dennis,  Cooper,  Valacich, 
Nunamaker  & Bastianutti,  1992).  According  to  the  study  carried  out  by  Dennis  (1993)  and  colleagues, 
electronic  communication  among  members  has  improved  the  idea-generation  performance  of  large  groups. 

Methods  of  Inquiry 

The  participants  in  this  study  were  the  pre -service  teachers  who  took  the  course  ‘Instructional 
Technology.’  In  this  course,  students  learn  how  to  effectively  integrate  IT  into  their  classroom  practices.  To 
create  a constructivist  learning  environment,  the  institution  adopted  Blackboard,  an  online  learning  delivery  and 
management  system  that  allows  students  to  learn  independently  and  instructors  to  customize  the  e-learning 
packages  according  to  their  students’  need. 

This  research  program  was  embarked  in  2001.  A class  of  twenty  pre -service  teachers,  each  with 
certain  characteristics  that  might  represent  the  target  population  was  selected  to  take  part  in  this  research 
program.  On  the  other  hand,  as  they  had  a common  teaching  subject  which  was  Chemistry  that  would  make  the 
discussions  more  subject-focused.  All  the  participants  were  given  WAP-enabled  mobile  handsets  for  their 
participation  in  the  online  discussions.  They  could  access  the  online  threaded  discussion  forum  using  the  given 
URL  anytime  and  anywhere  via  their  WAP-enabled  phones  or  the  web-based  forum  via  home  computers.  The 
participants  were  required  to  take  part  in  all  the  6 forums  which  consisted  of  3 case  studies  (with  one  as  a pilot 
test)  and  2 brainstorming  discussions. 

A pilot  test  was  carried  out  to  test  the  application  and  also  to  explore  the  implications  of  the 
discussions  generated  within  that  week.  During  this  pilot  test,  pre -service  teacher  and  their  course  instructor 
participated  in  the  case  study  online  discussions.  The  information  gathered  and  the  preliminary  analysis  was  of 
great  value  as  it  helped  to  identify  some  possible  obstacles  and  helped  to  refine  the  design  of  the  application. 
Also,  it  helped  to  refine  the  structure  and  content  of  the  discussion  forums. 

A survey  for  the  pre -service  teachers  was  conducted  at  the  end  of  all  6 forums  in  order  to  obtain 
statistical  evidences  and  also  to  generate  deeper  understanding  of  the  study.  A focus  group  interview  with  the 
pre -service  teachers  and  a face-to-face  interview  with  the  class  instructor  were  also  conducted  to  gain  an  in- 
depth  to  the  phenomena  surfaced  from  the  study;  the  transcripts  can  also  be  used  to  check  the  accuracy  of 
interview  records.  The  messages  posted  on  the  WAP-  and  WEB-based  forums  collected  in  the  form  of 
spreadsheet  were  then  translated  and  analyzed  into  simple  message  maps  that  showed  the  flow  of  messages. 

To  analyze  WAP-  and  WEB-based  discussions,  Jonassen’s  (2000)  rubrics  for  quality  discussions  and 
Jarvela  and  Hiikkinen’s  (2000)  classification  of  such  discussions  were  adopted  and  modified  in  this  study.  The 
messages  posted  in  the  both  forums  were  categorized  into  five  different  types  of  discussions  (suggestion, 
comment,  elaboration,  information-seeking  and  information-sharing)  under  four  criteria  with  three  levels  of 
participation  (high-level,  progressive-level  and  low-level).  These  four  criteria  included  accuracy,  relevance, 
coherence  of  the  messages,  and  the  levels  of  perspective  taking  of  messages.  Postings  that  were  irrelevant  to 
the  discussion  topics  were  considered  as  redundant  messages  and  were  not  analyzed. 

Findings 

The  result  of  the  survey  revealed  that  65%  of  the  pre -service  teachers  thought  that  brainstorming 
questions  were  more  suitable  for  WAP-based  discussions.  The  pre -service  teachers  felt  that  brainstorming 
questions  would  generate  more  new  ideas,  and  since  short  messages  were  needed,  it  would  be  easier  to  share 
ideas  in  a more  efficient  way.  The  other  35%  thought  that  case  study  questions  were  more  suitable  for  WAP- 
based  discussions  because  they  were  more  focused  than  brainstorming  questions.  The  results  suggested  that 
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majority  of  the  pre -service  teachers  thought  that  brainstorming  questions  were  more  suitable  for  WAP-based 
discussions.  However,  the  findings  from  the  messages  posted  on  the  forums  and  the  transcripts  from  the 
interviews  revealed  another  set  of  findings. 

The  percentages  of  messages  in  both  forums  were  low.  Quantitatively,  this  suggests  that  regardless  of 
whether  it  was  a brainstorming  discussion  or  a case  study  discussion,  it  does  not  affect  the  percentage  of  WAP- 
based  messages  posted  in  these  forums.  This  outcome  suggested  that  the  nature  of  brainstorming  discussions 
and  case  study  discussions  might  not  determine  the  discussion  outcome.  The  discussion  questions  might  throw 
some  light  on  this  phenomenon.  During  the  focus  group  and  face-to-face  interviews,  the  pre -service  teachers 
and  instructor  mentioned  that  the  topics  and  the  way  the  questions  were  phrased  did  determine  the  pre -service 
teachers’  responses.  One  pre -service  teacher  mentioned  : “I  think  it’s  the  discussion  topic  that  matters.  For  the 
discussion  on  school  experience. . .1  have  experience  to  share  with  them  (the  other  students)  so  I would  create  a 
thread,  if  not,  I would  just  see  what  people  have  to  say.”  Similarly,  another  pre -service  teacher  said:  “for  the 
school  experience,  I know  the  kind  of  environment  so  I can  comment  in  that  forum.”  Their  class  instructor  also 
emphasized  that  “the  factor  that  determines  the  participation  is  not  so  much  of  whether  it  is  case  study  or 
brainstorming,  rather  the  nature  of  the  discussion  question  itself.  If  the  nature  of  the  question  can  appeal  to 
them,  it  will  generate  greater  interest  and  thus  they  will  participate  more.”  This  might  explain  why  some  pre- 
service  teachers  preferred  brainstorming  discussions  while  others  found  that  case  study  discussions  more 
challenging.  The  more  the  students  could  relate  the  discussion  topic  to  their  own  personal  experiences,  the 
higher  the  level  of  interest  they  would  have  and  naturally  the  level  of  participation  would  increase. 

As  shown  in  table  1,  both  WAP-and  WEB -based  forums  produced  limited  number  of  high-level 
quality  messages.  However,  WAP-based  forums  had  yielded  more  low-level  quality  messages  and  less 
progressive-level  quality  messages  than  WEB-based  forums.  The  physical  and  technical  constraints  of  WAP- 
enabled  phones  might  not  have  supported  lengthy  messages.  Although  statistical  results  suggested  that  WAP- 
based  forums  have  produced  more  low-level  quality  messages  than  WEB-based  forums,  the  usefulness  and  the 
contributions  of  WAP  technology  should  not  be  ignored.  In  the  survey  that  was  conducted  at  the  end  of  the 
course,  65%  of  the  pre -service  teachers  agreed  that  WAP  technology  has  helped  to  build  a learning  community. 
They  believed  that  the  WAP-based  forum  has  formed  a closely  knitted  group,  and  everyone  was  able  to 
participate  and  learn  from  each  other. 


Quality  messages 
^Types^^ 

High-level  quality 
messages 

Progressive  -level 
quality  messages 

Low-level  quality 
messages 

WAP-based  forum 

6% 

29% 

56% 

WEB  -based  forum 

7% 

63% 

29% 

Table  1.  Percentages  of  Quality  Messages  for  WAP-based  and  WEB-based  Forums 


WEB-based  discussion  was  introduced  in  the  midst  of  forum  3.  The  introduction  of  the  WEB-based 
forum  encouraged  online  participation  and  it  did  not  negate  all  WAP-based  postings.  Five  students  continued  to 
visit  the  WAP-  and  WEB-based  forum  at  the  same  time.  This  suggested  that  the  WAP-based  forum 
complemented  the  WEB-based  forum.  The  pre -service  teachers  were  able  to  use  WAP-enabled  mobile  phones 
or  computers  to  participate  in  the  WAP-  or  WEB-based  forums.  In  this  case,  the  advantages  of  both  tools  could 
optimize  and  enhance  the  online  discussions.  In  other  words,  WAP  technology  provided  another  alternative  to 
the  online  discussions,  allowing  opportunities  for  further  collaboration  and  social  interaction.  During  the 
interview,  one  of  the  pre -service  teachers  commented  that  she  could  better  contribute  to  the  discussion  forums 
when  she  had  a choice  to  use  WAP-enabled  phone  if  she  was  on  the  move  or  use  her  computer  to  log  on  to 
WEB-based  forum  when  at  home.  The  other  pre -service  teachers  said:  “when  you  are  on  the  move,  at  least  you 
have  the  WAP-enabled  phone  to  view  the  unread  messages.”  Generally,  the  discussions  in  both  the  WEB-  and 
WAP-based  forums  were  rather  subject-focused,  content -related  and  constructive.  Irrelevant  messages  were 
minimal. 

The  language  that  was  used  in  the  WAP-based  forum  discussion  was  a unique  one.  It  was  a feature 
found  in  neither  formal  writing  nor  does  it  resembled  the  messages  found  in  the  WEB-based  forum.  In  WAP- 
based  message,  the  number  of  short  forms  used  was  more  frequent  and  perspective  taking  was  not  obvious. 
Although  statistical  results  suggested  that  WAP-based  forums  produced  more  low-level  quality  messages  than 
WEB-based  forums,  the  potential  and  the  contributions  of  WAP  technology  should  not  be  ignored.  In  the 
survey  that  was  conducted  at  the  end  of  the  course,  65%  of  the  pre  -service  teachers  agreed  that  WAP 
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technology  helped  to  build  a closely  knitted  community.  They  believed  that  the  WAP-based  forum  has  formed 
a closely  knitted  group,  and  everyone  was  able  to  participate  actively  and  learn  from  each  other.  By  looking 
into  the  language  pre -service  teachers  used  in  the  WAP-based  forum,  we  might  be  able  to  understand  why 
WAP-based  forum  helped  to  build  a closely  knitted  community.  Some  of  the  common  short  forms  that  were 
used  in  WAP-based  forums  were:  “stu”  for  “students”,  “n”  for  “and”  “chem”  for  “chemistry”  and  “2”  for  “to” 
etc.  For  someone  to  be  able  to  understand  the  syntax  of  these  messages , one  must  be  part  of  the  community 
long  enough  to  learn  how  to  represent  own  ideas  in  such  unique  ways  that  only  members  of  this  community 
could  understand. 

Another  interesting  phenomena  found  in  this  study  was  that  pre -service  teachers  and  class  instructor, 
adopted  different  roles  subtly.  In  many  other  situations,  pre -service  teachers  in  the  class  also  played  the  role  of 
a mentor  by  giving  constructive  suggestions  and  comments.  One  of  the  pre -service  teachers  created  threads  in 
order  to  better  guide  the  rest  when  discussing.  Although  their  class  instructor  did  not  assign  such  roles  to  the 
students,  they  have  adopted  diverse  roles  subtly  to  facilitate  their  online  discussions.  The  majority  of  the  pre  - 
service teachers  also  agreed  that  the  class  instructor  played  a crucial  role  in  the  online  discussions  and  that  he 
had  fulfilled  his  job  as  mentor,  guidance  and  listener. 


Conclusion 

Learners’  changing  characteristics  prompt  us  to  look  into  other  new  modes  of  course  delivery.  This 
study  revealed  the  potential  of  WAP  technology  as  an  effective  online  communicating  tool  when  coupled  with 
other  tool.  It  also  documented  the  pioneering  efforts  of  using  WAP  technology  in  the  online  discussions.  More 
studies  are  needed  to  explore  the  possible  ways  of  making  WAP  technology  a successful  social  and  intellectual 
tool  for  mediating  individual  learning  and  enhancing  the  social  construction  of  knowledge. 


References 

Althaus,  S.  C.  (1997).  Computer-mediated  communication  in  the  university  classroom:  An  experiment  with 
online  discussion’,  Communication  Education,  46,158-174. 

Althauser,  A.,  & Matuga,  J.  M.  (1998).  On  the  pedagogy  of  electronic  instruction.  In  Bonk,  C.  J.  and  King,  K.S. 

(Eds.),  Electronic  collaborators:  Learner-centered  technologies  for  literacy,  apprenticeship,  and  discourse 
(pp. 183-208).  Mahwah:NJ:  Lawrence  Erlbaum  Associates 

Bonk,  C.  J.,  & Cunningham,  D.  J.  (1998).  Searching  for  learner-centered,  constructivist,  and  sociocultural 
components  of  collaborative  educational  learning  tools.  In  Bonk,  C.  J.,  & King,  K.  S.  (Eds.), 
Electronic  collaborators:  Learner-centered  technologies  for  literacy,  apprenticeship,  and  discourse 
(pp.25-50).  Mahwah,  New  Jersey,  Lawrence  Erlbaum  Associates. 

Box,  K.  (1999).  Human  interaction  during  teacher  training  courses  delivered  via  the  internet.  Texas,  United 
States  of  America:  University  of  North  Texas.  (ERIC  Document  Reproduction  Service  No.  ED432 
224). 

Chong,  S.  M.  (1998).  Models  of  asynchronous  computer  conferencing  for  collaborative  learning  in  large 
college  classes.  In  Bonk,  C.  J.  and  King,  K.S.  (Eds.),  Electronic  collaborators:  Learner-centered 
technologies  for  literacy,  apprenticeship,  and  discourse  (pp. 157- 182).  Mahwah:NJ:  Lawrence 
Erlbaum  Associates . 

Curtis,  D.  D.,  & Lawson,  M.  J.  (2001).  Exploring  collaborative  online  learning,  JALN,  5,  21-34. 

Dennis,  A.R.,  & Valacich,  J.S.  (1993).  Computer  Brainstroms:More  heads  are  Better  Than  One,  Journal  of 
Applied  Psychology,  78,  531-536. 

Gallupe,  R.  B.,  Bastianutti,  L.  M.,  & Cooper,  W.  H.  (1991).  Unblocking  Brainstorms , Journal  of  Applied 
Psychology.  76,137-142. 

Gallupe,  R.  b.,  Dennis,  A.  R.,  Cooper,  W.  H.,  Valacich,  J.  S.,  Nunamaker,  J.  F.,  Jr.,  & Bastianutti,  L.  (1992). 
Electronic  brainstorming  and  group  size.  Academy  of  Managment  Journal.  35,350-369. 

Gunawardena,  C.  N.,  & Msisaac,  S.  (2004).  Distance  Education.  In  Jonassen,  D.  H (Eds.)  Handbook  of 
research  on  educational  communications  and  technology,  pp355-396.  Mahwah:  NJ:  Lawrence 
Erlbaum  Associates. 

Jarvela,  S.  and  Hakkinen,  P.  (2000,  in  press).  Web-based  cases  in  teaching  and  learning-the  quality  of 

discussions  and  a stage  of  perspective  taking  in  asynchronous  communication.  Interactive  learning 
Environments. 

Jonassen,  D.H.  (2000),  Computers  as  Mindtools  for  Schools:  Engaging  Critical  Thinking.  Columbus,  OH, 
Prentice-Hall. 


525 


King,  K.  S.  (1998).  Designing  21st-century  educational  networlds:  Structuring  electronic  social  spaces.  In 

Bonk,  C.  J.  and  King,  K.S.  (Eds.),  Electronic  collaborators:  Learner-centered  technologies  for  literacy, 
apprenticeship,  and  discourse  (pp. 365-384).  Mahwah:  NJ:  Lawrence  Erlbaum  Associates. 

Lapadat,  J.  C.  (2000).  Teaching  Online:  Breaking  New  Ground  in  Collaborative  Thinking,  Paper  presented  at 
the  Annual  Meeting  of  the  Canadian  society  for  the  Study  of  Education.  Canada,  May. 

Lee.C.  B.  (2002).  Exploring  WAP  technology  to  enhance  online  discussion.  Unpublished  master’s  thesis, 
Nanyang  Technological  University,  Singapore. 

Rogoff,  B.  (1990).  Apprenticeship  in  Thinking:  Cognitive  Development  in  Social  Context.  New  York:  Oxford  : 
University  Press. 

Salomon,  G.  (1993).  On  the  nature  of  pedagogic  computer  tools:  The  case  of  the  writing  partner.  In  S.P. 

Lajoie,  & S.  J.  Derry  (Eds.),  Computers  As  Cognitive  Tools,  (pp  179- 18 6)  Hillsdale:  NLLawrence 
Erlbaum  Associates . 

Zhu,  E.  (1998).  Learning  and  mentoring:  Electronic  discussion  in  a distance  learning  course.  In  Bonk,  C.  J.  and 
King,  K.S.  (Eds.),  Electronic  collaborators:  Learner-centered  technologies  for  literacy, 
apprenticeship,  and  discourse  (pp.233-260).  Mahwah:NJ:  Lawrence  Erlbaum  Associates . 


526 


A Learning  Process  in  Resource -Based  Well-Structured  Instruction  in 
Web-based  Distance  Learning  Environment 


Hye-Jung  Lee 
Seoul  National  University 

Byeong-Min  Yu 
Konkuk  University 


Abstract 

This  research  is  to  articulate  a learning  process  in  resource-based  well-structured  instruction  in  web- 
based  distance  learning  environment.  To  explore  the  learning  process,  a web  resource-based  self-learning 
program,  which  is  highly  structured  with  structuring  elements  from  literature  review,  was  developed  as  a 
college  level  course.  Sixty-seven  junior  students  at  a cyber  university  in  Korea,  aged  from  20s  to  50s,  were 
required  to  learn  the  program  given  for  eight  weeks.  Students’  perceived  learning  processes  from 
Questionnaires,  achievement  scores,  satisfaction  levels,  interview  data,  were  collected  and  analyzed.  As  a 
result,  an  observable  action  model  and  a conceptual  model  of  the  learning  process  were  derived,  which  students 
and  experts  verified.  Cognitive  achievement  factor  and  satisfaction  factor  were  also  considered  in  the  learning 
process  model.  Consequently  the  learning  process  constitutes  a major  phase  (initiation  ? contact  learning 
material  ? interned  process  ? individual  understanding  ? externedization  ? internalization)  and  a minor 
phase  (minor  informed  offline  interaction  among  students ) in  resource-based  highly  structured  self-learning. 
Implications  of  each  stage  in  the  model  and  recommendations  for  further  study  were  suggested. 

Context  of  the  Research 

Web  and  multimedia  technology  is  changing  the  way  of  teaching  and  learning.  There  is  also  an 
increasing  demand  for  a flexible  learning  framework  that  does  not  tie  the  learner,  especially  adult  learner  with 
full  time  job,  down  to  a specific  time  or  place.  This  is  the  very  need  for  distance  education.  In  distance 
education,  people  cannot  interact  well  because  of  the  ‘distance’.  For  thousands  of  years,  learning  and  teaching 
always  took  place  in  close  proximity,  and  this  has  become  firmly  anchored  in  human  consciousness.  Learning 
and  teaching  at  a distance  has  been  therefore  regarded  as  something  extraordinary  and  as  a defect.  Because  of 
the  distance,  educators  have  tried  to  make  every  effort  to  get  over  this  spatial  separation.  The  first  pedagogic 
approaches  specific  to  distance  education  aimed  at  finding  ways  by  which  spatial  distance  could  be  bridged, 
reduced  or  even  eliminated.  The  question  was  asked  (Peters,  1998):  what  must  be  done  to  make  distance 
equivalent  to  proximity  in  distance  education?  The  pedagogics  of  distance  education  is  derived  basically  from 
the  efforts  to  answer  to  this  question. 

One  of  the  try  to  answer  to  this  question  is  making  and  managing  learning  materials  as  best  ever  as 
possible,  which  almost  does  not  need  teacher’s  interaction.  Educators  at  traditional  distance  university  used 
most  funds  and  most  efforts  for  the  professional  development  and  production  of  qualitatively  excellent  teaching 
materials  for  the  purposes  of  self-study,  which  are  then  distributed  by  post  in  early  distance  education  and  by 
web-based  cyber  instruction  program  in  these  days.  They  tried  to  include  teacher’s  interaction  into  the  well- 
structured  material  for  self-learning.  Moore(1993)  conceptualized  these  characteristics  as  ‘structure’,  because  he 
regarded  the  main  characteristic  of  the  making  and  managing  programs  as  the  structuring  of  the  learning  and 
teaching.  Also  researches  have  presented  elements  and  rationale  of  course/contents  structure  as  well  as 
instructional  strategies  for  structuralization.  Researchers  emphasizing  structure  in  distance  learning  seem  to 
believe  that  well-structured  material  can  get  over  the  absence  of  teacher.  Based  on  this  belief,  educators  in 
Korea  have  provided  mainly  structure  dependent  program  a lot,  which  are  for  self-learning  with  little 
interaction. 

Unfortunately,  however,  it  is  not  easy  to  find  how  the  structure  influences  on  learning  process. 
Although  majority  of  the  literature  on  web-based  distance  education  focuses  on  the  effectiveness  of  web-based 
distance  education  or  educational  medium,  and  emphasizes  structuring  (converting  existing,  traditional  course 
materials  to  a wed-based  format),  they  don’t  show  how  student’s  learning  process  is  going  in  highly  structured 
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instruction. 

Therefore,  the  purpose  of  this  research  is  to  articulate  the  learning  process  in  resource-based,  highly 
structured  instruction  in  web-based  self-learning  environment.  The  learning  process  model  from  this  research 
will  be  able  to  explain  how  students  learn  with  highly  structured  web  materials  and  how  course  and  contents 
structure  influence  on  distance  learning.  And  it  will  also  provide  better  understanding  of  structural  instructional 
strategies  ’ mechanisms  as  well  as  practical  implications  for  distance  instructional  program  design. 

Literature  Review 

There  is  physical  separation  of  teacher  and  learner,  which  is  ‘distance’,  in  distance  education.  One  of 
the  efforts  of  reducing  the  distance  between  teachers  and  students  was  taken  when  people  started  to  interpret 
distance  education  as  the  simulation  of  a conversation  between  teacher  and  student.  They  recommend  that  the 
contents  be  displayed  in  the  form  of  a written  dialogue.  When  writing  teaching  texts,  teachers  must  imagine  that 
they  are  speaking  to  someone,  and  this  is  supposed  to  make  them  use  a spoken  language  wherever  possible.  The 
way  in  which  the  contents  are  shown  must  enable  students  to  imagine  the  teacher  in  person  while  they  are 
reading  and  to  carry  on  quiet  dialogues.  Reading  teaching  texts  and  assimilating  their  contents  is  thus 
transformed  into  an  internal  or  virtual  dialogue  (Holmberg,  1983). 

The  term  of  ‘structure’,  Moore  (1993)  defined,  is  contrast  to  dialogical  learning.  It  is  consistently 
planned  on  a targeted  basis  and  with  small  steps,  therefore  being  closed  to  spontaneous  interventions,  its  time  is 
regimented  and  it  is  uniformly  controlled  and  evaluated.  The  main  instruments  of  this  learning  and  teaching  are 
printed  courses  or  multimedia  learning  packages  which  contain  carefully  developed  and  optimized  courses  and 
set  learning  into  motion  and  control  it.  Moore(  1993)  chose  the  brief  description  of  ‘structure  ’ for  this  concept, 
because  he  regarded  the  main  characteristic  as  the  structuring  of  the  learning  and  teaching  - right  down  to  the 
last  detail.  He  used  the  example  of  a teaching  film  for  television  to  show  just  how  far  this  structuring  could  go: 
in  the  film  ‘literally  every  word,  every  action  of  the  teacher,  every  minute  of  the  available  time  and  even  the 
tiniest  detail  of  the  contents  were  laid  down  beforehand’.  This  kind  of  reduction  of  structuring  program  is  on  the 
basis  of  criteria  from  educational  technology.  The  structure  is  seen  most  clearly  and  most  densely  in 
programmed  instruction.  It  was  the  representative  of  an  instructional  theory  in  which  each  word,  each  learning 
step,  each  teaching  strategy  was  planned  and  developed  with  the  greatest  care  in  order  to  simplify  learning  and 
to  make  teaching  more  successful. 

The  structuring  of  learning  and  teaching  is  in  fact  nothing  new  and  certainly  not  unique  to  distance 
education.  Presenting  knowledge  through  books  has  usually  been  broken  down  by  means  of  components  such  as 
a foreword,  an  introduction,  chapters,  a summary  and  a conclusion.  And  in  the  last  two  centuries,  the 
articulation  of  instruction  has  been  a central  theme  of  pedagogics,  especially  in  the  field  of  instructional  design. 
What  is  nevertheless  new  in  Moore’s  suggested  concept  is  not  structuring  in  itself  but  the  extreme  extent  of  its 
application  (Peters,  1998). 

This  ‘structure’  discussed  here,  is  about  the  elements  in  the  course  design  and  contents  design.  A 
course  consists  of  such  elements  as  learning  objectives,  content  themes,  information  presentations,  case  studies, 
pictorial  and  other  illustrations,  exercises,  projects,  tests.  All  these  may  be  very  carefully  composed,  very 
carefully  structured.  In  addition,  we  need  to  consider  another  type  of  structure  especially  in  web-based 
environment,  which  is  interface  structure.  Interface  structure  is  related  to  site  structure,  navigation  structure,  or 
screen  design  structure  on  the  screen  that  students  face  directly(Lee,  2004b).  Therefore,  highly  structured 
instruction  in  this  research  means  that  the  extent  of  structuring  is  quite  high  in  terms  of  course,  contents,  and 
interface  structure  for  self  learning  without  teacher’s  interaction.  Structural  elements  in  detail,  modified  from 
Lee(2004a)  and  Peters(1998),  are  shown  in  Table  1. 

Table  1.  Modified  structural  elements  (Lee,  2004a;  Peters,  1998) 

Structure  Structural  elements 

Dosing 

Course  orientation:  welcoming,  characterizing  the  course  contents,  description  of  the  required 
preliminary  knowledge,  indications  of  correlations  to  other  areas  of  knowledge  and  possibly  to  job 
practice,  description  of  rough  learning  and  teaching  objectives,  naming  advantages  obtained  by 
reaching  and  learning  goals . . . 

Scheduling 
Self-tests 
Peer  evaluation 

• Assignments  and  pre-scheduled  deadline 


Course 

structure 
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Contents 

structure 


Interface 

structure 


Systematic  management  of  course  pace  and  learning  speed 
Strict  grading  according  to  notice  announced  in  advance 
Portioning 
Contents,  lists 

Indication  of  learning  and  teaching  aims 
Sequencing 

Stressing  what  is  important 

Showing  perspectives  for  further  study 

Branching 

Cross-references 

Arrangement  in  blocks  and  characterization  by  marginal  notes 

Summative  map,  structural  site  map,  procedural  map 

Concept  map,  concepts  tree 

Comparison/Contradiction 

Abstracts 

Index 

Instructional  methods:  questions  for  repetition,  questions  for  training,  simulated  dialogue 

Writing  contents  with  spoken  language,  like  conversation  style 

Spatial  layout  to  avoid  cognitive  overloading 
Search  index  function 
Emphasizing  by  spaced  printing 
Marginal  notes 

Visualization  with  graph,  table,  or  figures 

Frames,  underlining,  bold,  italic  print,  decimal  numbering 

Use  characters  like  avatar  for  students  not  to  feel  isolation 


Research  Method 

In  order  to  articulate  the  learning  process  in  structured  distance  instruction,  we  developed  a resource- 
based  highly  structured  web-based  distance  instructional  program  as  college  level.  The  program  was  developed 
and  implemented  mainly  with  resource-based  highly  structured  self-learning  mode.  Structuring  elements  in 
course/contents/interface  from  literature  review  (Lee,  2004a;  Peters,  1998)  were  applied  to  the  program.  We 
designed  and  developed  the  instructional  program  according  to  traditional  ISD( Instructional  Systems 
Development)  model,  considering  structural  elements  from  the  literature.  Three  WBI(Web-Based  Instruction) 
experts  and  practitioners  verified  the  program  on  and  off  throughout  the  development  process.  The  subject  of 
the  program  was  about  general  social  science. 

Sixty-seven  junior  students  at  K Cyber  University  in  Korea,  aged  from  20s  to  50s,  randomly  assigned, 
were  required  to  study  highly  structured  web  material  for  eight  weeks  and  to  undertake  assignments  given  every 
week.  Instructor’s  feedback  was  provided  as  little  as  possible.  Instead,  most  possible  instructor’s  feedbacks 
were  structured  into  the  web  materials.  The  instructor  was  trying  to  neither  encourage  nor  discourage 
interactions  among  students.  Throughout  the  course,  students  ’ perceived  learning  process  from  questionnaires, 
cognitive  achievements,  satisfaction  levels,  online  messages,  interview  data,  and  participatory  investigation  data 
were  collected  and  analyzed.  Various  statistical  analysis  methods  such  as  correlation  analysis,  contents  analysis, 
t-test,  frequency  analysis,  were  applied  to  the  data. 

For  evaluation  reliability,  three  evaluators’  gradings  were  correlated  (Pearson  r = .84  ).  In  terms  of 
satisfaction  level,  a satisfaction  measurement  tool  (Kim  & Ryu(2000)  were  developed  and  validated)  was  used 
after  modification  (reliability  alpha  = .93).  Twenty  students  were  interviewed  at  the  end  of  each  course  to  verify 
all  quantitative  data  and  to  provide  more  detailed  information  to  the  researchers  regarding  the  factors  of  learning 
process. 

Modeling  procedure  of  learning  process  was  following  Rubinstein  (1975)’s  perspective.  According  to 
him,  modeling  procedure  is  for  simplifying  and  abstraction.  So  the  procedure  in  this  research  was  as  follows: 
Development  of  treatment  instructional  program  ? Implementation  of  the  program  and  collecting  data  ? 

Coding  questionnaire  data  ? Deriving  rough  pattern  of  learning  process  ? Analysis  of  learning  output 
variables  ? Correlation  of  learning  output  variables  and  learning  process  ? Interview  ? Analysis  of  interview 
data  ? Verification  learning  process  action  model  by  learners  ? Conceptualization  of  learning  process  and  its 
visualization  ? Verification  of  conceptualized  learning  process  model  by  experts  and  learners  ? Production  of 
finally  verified  conceptual  diagram  of  learning  process  in  resource-based  well-structured  WBI  learning 
environment. 
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Results  and  Discussion 


Learning  process;  an  action  model 

To  explore  a learning  process  in  resource-based  structured  distance  self-learning,  we  coded  sixty-seven 
students  ’ perceived  learning  processes  in  ques  tionnaires  and  derived  rough  average  pattern  of  the  students  ’ 
learning  process.  The  students’  perceived  learning  procedure  after  first  coding  was  composed  of  1 1 stages  as 
follows:  reading  notice  & information  provided,  reading  free  board,  reading  Q/A  board,  posting  messages  on 
free  board  or  Q/A  board,  studying  web-based  material,  editing  and  printing  web  text,  doing  assignments, 
making  reference  to  other  materials  through  on/off  line,  off  line  interaction  (telephone  or  offline  meeting), 
assignments  submission,  check  individual  learning  pace  in  learning  management  system.  Among  these  items, 
activities  less  than  10%  frequency  were  removed,  and  the  learning  procedure  was  recoded  again.  Also  all  stages 
were  analyzed  and  correlated  with  learning  output  variables  such  as  cognitive  achievement  or  satisfaction. 
Throughout  this  procedure,  a learning  process,  in  which  all  stages  were  rearranged  with  logical  sequence,  was 
finally  derived  (see  Figure  1).  Figure  1 shows  an  observable  action  model  of  learning  process  in  resource-based 
highly  structured  instruction  in  web-based  distance  learning  environment.  Subscript  1)  represents  cognitive 
achievement  factor  and  subscript  2)  indicates  satisfaction  factor. 


Figure  1.  An  observable  action  model  of  learning  process  in  resource-based  highly  structured  instruction  in 
web-based  distance  learning  environment;  1)  Cognitive  achievement  factor,  2)  Satisfaction  factor,  3) 

Interactions  (Students  -Contents(S-C),  Students -Students(S-S)). 

The  learning  process  in  this  research  constitutes  a major  phase  and  a minor  phase.  Major  phase  is  about 
individual  learning  and  the  minor  phase  of  the  model  is  about  minor  informal  offline  interaction  among 
students . In  major  learning  process,  students  [contact  web  material],  which  is  highly  structured,  and  then  are 
involved  in  [undertaking  assignments].  After  the  process  of  their  assignments  of  each  web  lecture,  students 
finally  [produce  results].  While  undertaking  assignments,  some  students  (33%)  informally  interacted  with  other 
colleagues.  This  interaction  was  offline  out-of-class  communication,  by  telephone  or  face-to-face  meeting. 
Although  asynchronous  bulletin  boards  were  provided  during  the  course,  students  ’ interaction  was  not  so 
activated.  Instead,  students  interacted  informally  outside  of  class,  mainly  about  social  or  procedural  topics,  far 
from  academic  discussion.  This  interaction,  therefore,  didn’t  seem  to  be  main  learning  process.  Interviewee 
indicated  that  students  studied  alone  but  sometimes  they  were  hungry  for  social  relationship  with  classmates, 
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especially  in  distance  learning  environment. 

One  of  the  significant  findings  in  this  model  is  that  the  stage  of  [print  web  transcripts]  is  found  to  be  a 
learning  achievement  factor  (p<  .01).  Students  who  printed  web  material  showed  higher  achievement  score  than 
those  who  didn’t  print.  The  interviewee  said  that  they  printed  web  materials  because  it  was  easier  to  read 
printed  text  on  paper  than  to  read  the  text  on  the  screen,  and  it  is  easy  to  take  a note  on  it  what  they  learned. 
Apparently  this  could  mean  that  reading  texts  on  paper  would  be  more  effective  that  reading  texts  on  screen. 

But  the  effectiveness  in  this  context  could  be  from  well -structured  contents  in  the  material,  not  from  printed 
format  itself.  That  is,  structuring,  rather  than  printing,  could  be  main  reason  for  the  effectiveness.  So  we  suggest 
that  the  exact  critical  success  factor  in  achievement  be  further  researched.  Anyhow,  printing  activity  in  this 
research  was  used  for  studying  more  intensely  and  for  externalization  of  what  they  understand. 

In  terms  of  satisfaction  level,  the  structure  of  course  and  contents  was  found  to  be  a critical  factor. 
Upper  30%  of  achievement  group  showed  significantly  higher  satisfaction  on  the  course  and  contents  structure 
than  lower  30%  group  showed  (p<  .01). 

Learning  process;  a conceptual  model 

The  action  model  of  learning  process  was  abstracted  to  a conceptual  model  (see  Figure  2),  with 
considering  previous  literature  (Lee,  2004a;  Stahl,  2000).  In  Figure  2,  bold  solid  arrow  shows  major  learning 
process  and  fine  solid  arrow  shows  back  process  or  minor  process  experienced  by  some  students.  Dotted  line 
represents  partial  experience  of  some  students,  not  a major  process. 


Figure  2.  A Conceptual  model  of  learning  process  in  resource-based  well-structured  instruction  in  web- 
based  distance  learning  environment 


In  the  conceptual  diagram  of  learning  process,  there  are  two  phases.  One  is  about  ‘individual  learning’  and  the 
other  is  about  ‘social  interaction’,  which  is  differentiated  from  ‘social  learning’.  In  individual  learning  phase, 
the  process  is  beginning  with  [initiation]  at  first.  [Initiation]  includes  access,  reading  notice  or  announcement, 
clicking  menu,  exploration  on  web-based  class  sites.  This  activity  is  not  exactly  main  learning  process,  rather 
pre -activity  before  learning  process.  So  it  is  shown  outside  of  the  learning  cycle. 

After  initiation,  learners  [contact  learning  materials],  which  were  highly  structured  web-based 
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instructional  program  in  this  research,  printed  materials  of  web-based  instructional  program,  and  other 
resources.  At  this  point,  students  printed  web  materials  for  reading  more  intensely  and  for  note -taking.  Making 
printed  material  was  found  to  be  a critical  achievement  factor  and  satisfaction  factor  in  this  research  as 
mentioned  in  Action  model  section. 

When  learners  contact  learning  material,  there  must  be  [internal  process]  in  their  brain,  which  is  many 
learning  theorists  have  concerned  about.  At  this  stage,  students  are  trying  to  absorb  knowledge  into  their  own 
cognitive  schema  over  reflection.  This  is  occurred  in  one’s  brain  for  self -asking  or  intrapersonal 
communication,  which  is  not  observable  but  explainable  by  learning  theories  of  cognitive  scientists.  But  we 
don’t  discuss  how  the  internal  process  is  going  on  here  because  it  is  beyond  this  research. 

If  students  understand  the  contents  well  enough  throughout  the  internal  process,  they  arrive  at 
[individual  understanding].  Otherwise,  they  get  back  to  [contact  learning  material]  and  repeat  this  cycle  until 
they  understand  it.  [Individual  understanding]  is  differentiated  from  "shared  understanding’  presented  by  Stahl 
(2002).  Whereas  ‘socially  shared  understanding]  is  from  consensus  among  interactive  team  members, 

[individual  understanding]  is  the  arrival  stage  of  the  process  from  individual  learner’s  cognitive  structure 
through  internal  process  to  resolving  cognitive  conflicts.  This  [Individual  understanding],  which  is  temporary 
understanding  but  not  externalizable,  is  also  different  from  final  [internalization]  that  is  personal  comprehension 
of  knowledge. 

To  get  to  [internalization],  it  is  found  that  students  summarize  or  take  notes  of  what  they  understand. 
This  activity  is  conceptualized  as  [externalization]  in  this  research,  which  is  found  as  a critical  cognitive 
achievement  factor.  The  observable  behavior  in  this  step  was  printing  web-text  and  utilizing  it.  This  means  that 
physical  behavior  of  printing  represents  significant  cognitive  activity  in  learning  process. 

After  this  whole  cycle,  a learner  eventually  arrives  at  [internalization]  that  is  personal  comprehension 
of  knowledge.  Then  you  have  to  decide  to  continue  or  stop  learning.  This  whole  process  can  be  visualized  as  a 
conceptual  diagram  of  learning  process  (see  Figure  2).  The  conceptual  model  presented  in  Figure  2 was  verified 
by  five  distance  education  experts  and  10%  of  students  in  this  research.  Respondents  used  a 5-point  Likert  scale 
(5  = fully  verified,  1 = not  verified),  which  was  referred  to  previous  literature  (Choi,  2002;  Rha  & Hong,  2004; 
Rha  & Jung,  2001),  to  rate  validity,  explicability,  usability,  generality,  and  comprehensibility.  Average  rate  of 
experts  was  4.28  and  average  rate  of  students  was  4.10. 

Conclusion  and  Recommendation 

This  research  is  to  articulate  the  learning  process  in  resource-based  well-structured  instruction  in  web- 
based  self-learning  environment.  To  explore  the  learning  process,  a resource-based  self-learning  program  was 
developed.  Quantitative  and  qualitative  data  were  collected  from  the  implementation  of  the  program  and  an 
action  model  and  a conceptual  model  of  learning  process  were  derived.  In  conclusion,  the  learning  process  in 
resource -based  structured  distance  learning  constitutes  a major  phase  (initiation  ? contact  learning  material  ? 
internal  process  ? individual  understanding  ? externalization  ? internalization)  and  a minor  phase  (minor 
informal  offline  interaction  among  students). 

The  findings  in  this  study  provide  some  significant  implications.  One  is  that  this  study  provides  a 
conceptual  framework  to  understand  how  the  learning  process  is  going  in  highly  structured,  web  resource  based 
distance  instruction  with  little  interpersonal  interaction.  This  article  also  contributes  to  understand  how  the 
course  and  contents  structure  influence  in  distance  learning.  The  other  is  that  we  can  suggest  some  instructional 
strategies  in  self-learning  mode,  such  as  externalization  strategies  or  design  of  appropriate  social  interaction. 
Printing  and  note-taking  turns  out  to  be  a critical  success  factor  in  this  research.  So  externalization  tips  like 
trying  to  require  students  to  submit  reflection  note  for  assignments  could  be  a good  strategy  for  better  cognitive 
achievement.  Also  social  interaction  among  students  needs  to  be  encouraged  in  especially  distance  learning 
environment.  We  may  design  this  to  be  structured  in  a course.  Historically  educators  have  tried  to  include  most 
possible  student -teacher  interaction  into  a learning  material.  But  findings  in  this  research  show  that  now  we 
need  to  consider  structuring  student -student  interaction,  as  well  as  student-teacher  interaction,  in  instructional 
design  of  web-based  distance  learning  program. 

Finally  recommendations  for  further  research  to  get  over  limitations  in  this  research  are  suggested  as 
follows:  a)  This  research  is  a case  study,  based  on  Korean  context.  There  could  be  difference  in  other  cultural 
context.  So  we  suggest  that  similar  research  in  different  context  as  well  as  in  different  learning  subject  be 
studied  to  enrich  the  implications  in  this  article.  Besides,  learning  process  study  in  mainly  interpersonal 
interactive  learning  environment  or  their  comparative  study  can  be  proposed  for  further  research.  These 
researches  providing  quantitative  data  from  more  cases  with  various  learning  contexts  could  verify  and 
generalize  the  findings  of  this  research.  Also  researches  considering  emotional  or  social  evaluation  as  well  as 
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cognitive  evaluation  on  various  learning  contents  could  be  recommended  for  more  comprehensive 
understanding  in  web-based  distance  education,  b)  This  research  did  not  consider  each  structural  element’s 
effect  or  influence  on  learning,  although  many  structural  elements  from  the  previous  literature  were  applied  to 
structured  instructional  development.  But  there  could  be  many  different  design  types  among  structured 
instructional  program.  Analysis  and  comparison  of  each  element’s  influence  on  learning  would  be  a good  theme 
for  further  research,  c)  Learner  analysis  in  each  group  might  contribute  to  elaborate  the  implications  in  this 
research.  We  didn’t  analyze  learners’  characteristics  with  objective  measurement  tools.  Rather  we  simply 
checked  perceived  learners’  characteristics.  More  objective  analysis  of  learners’  characteristics  with  verified 
measuring  tools,  including  special  characteristics  such  as  learner  autonomy  or  self -regulating  ability,  would 
give  us  significant  and  more  articulated  implications. 
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Analysis  of  essential  skills  and  knowledge  for  teaching  online 
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Abstract 

This  study  identifies  and  posits  guidelines  for  assessing  the  skills  and  knowledge  of  online  distance 
educators.  Findings  derived  from  Cooper’s  (1998)  synthesis  research  method  reveal  sixteen  skills  that  may  be 
grouped  into  six  areas  that  are  thought  to  be  essential  for  educators  to  teach  successfully  online.  The  study  also 
shows  that  factors,  like  discipline,  delivery  mode,  learning  outcome,  and  instructional  strategy,  may  affect  the 
application  of  those  skills,  Moreover,  the  findings  suggest  that  educators  can  greatly  benefit  from  training, 
support,  and  faculty  development  to  make  the  transition  from  teaching  in  a face-to-face  setting  to  an  online 
setting,  even  though  most  skills  are  thought  to  be  similar  and  applicable  across  settings.  The  findings  also 
indicate  that  further  studies  are  needed  to  establish  the  validity  and  reliability  of  self-assessment  instruments 
and  to  connect  theory  and  practice. 


Introduction 

Online  distance  learning  has  become  a very  popular  mode  for  learning  in  higher  education  during 
recent  years.  However,  some  online  educators  claim  that  they  were  forced  to  teach  online  (Cooper,  1998),  or 
they  lack  the  knowledge  or  skills  to  teach  successfully  online  (Bailey  & Chambers,  1996;  Clark,  1993; 
Flottemesch,  2000;  Inman  & Kerwin,  1999;  Karsenti,  2001;  Wallace,  2003).  Do  distance  educators  need  any 
special  skills  or  knowledge  to  teach  effectively  online?  If  so,  what  are  they?  Moreover,  is  there  any  valid, 
reliable,  and  efficient  way  to  determine  whether  an  educator  is  suitable  for  teaching  online? 

The  purposes  of  this  study  are  to  (a)  determine  whether  online  distance  educators  need  special  skills  or 
knowledge  to  successfully  teach  online,  and  (b)  determine  whether  any  valid,  reliable  and  efficient  ways  to 
assess  the  essential  skills  and  knowledge  of  online  distance  educators. 

This  study  synthesized  the  skills  and  knowledge  for  online  distance  educators  from  a review  of 
literature  and  an  analysis  of  existing  self-assessment  instruments.  The  research  questions  that  guided  the  study 
were : 

1.  What  are  essential  skills  of  a successful  online  distance  educator  in  higher  education? 

2.  Is  there  difference  between  teaching  in  face-to-face  and  online  environment? 

3.  Is  a self-assessment  available  for  educators  to  determine  their  capability  and  suitability  for 
teaching  online? 

4.  What  elements  in  existing  self-assessment  instruments  were  used  in  higher  education? 

5.  Is  there  an  alternative  way  to  determine  whether  an  educator  in  higher  education  is  suitable  for 
online  teaching? 

For  the  purposes  of  this  study,  Hirumi ’s  (2002)  definition  fore-learning  is  used  to  describe  online 
distance  learning  that,  is  facilitated  predominately  through  the  use  of  telecommunication  technologies  such 
as  electronic  mail,  electronic  bulletin  board  systems,  inter-relay  chat,  desktop  videoconferencing  and  the  World- 
Wide-Web’'(p.l7).  Online  courses  are  increasing  dramatically  across  higher  education.  According  to  the 
Statistics  from  U.S.  Department  of  Education,  National  Center  for  Education,  the  percentage  of  all  2-year  and  4- 
year  institutions  that  provided  distance  education  courses  for  any  level  or  audience  doubled  from  1997-98  to 
2000-2001  academic  year  (Lewis,  Alexander,  Farris,  & Greene,  1997;  Livingston  & Wirt,  2004;  Waits  & 

Lewis,  2003).  However,  online  distance  learning  also  presents  issues. 

The  problem  is  the  overall  quality  of  online  courses  is  still  questionable  (Johnson,  2003).  Even  though 
many  educators  see  the  importance  of  online  education,  many  still  lack  the  experience  or  knowledge  to  teach 
online  effectively  (Bailey  & Chambers,  1996;  Clark,  1993;  Flottemesch,  2000;  Inman  & Kerwin,  1999; 

Karsenti,  2001;  Wallace,  2003).  Do  educators  need  any  special  skills  or  knowledge  to  successfully  teach  online? 
If  so,  are  there  any  effective  and  efficient  ways  to  measure  or  determine  if  an  educator  has  the  skills  and 
knowledge  necessary  to  teach  effectively  online? 

Published  studies  use  different  techniques  to  delineate  essential  skills  for  distance  educators.  A number 
of  studies  use  Delphi  techniques  to  identify  roles  or  skills  that  distance  educators  should  possess  (Thach,  1994; 
Williams,  2003).  Some  studies  use  quantitative  methods  to  examine  what  educators  face  and  feel  about  online 
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teaching  (Clark,  1993;  Inman  & Kerwin,  1999).  Others  focus  on  the  shift  educators  make  from  teaching  in  a 
face-to-face  environment  to  that  in  an  online  environment  (Conciecao-Runlee,  2001;  Coppola,  Hiltz,  & Rotter, 
2002;  Easton,  2003;  G.  G.  Smith,  Ferguson,  & Caris,  2002).  Many  authors  expressed  concern  for  terminology 
in  the  field  (Easton,  2003;  Flottemesch,  2000).  To  gain  insights  and  a better  understanding  of  the  research 
results,  it  is  helpful  to  look  at  the  “big’'  picture  by  synthesizing  findings  from  previous  studies. 

Method 

This  study  used  the  research  synthesis  method  developed  by  Cooper  (1998)  to  identify  fundamental 
skills  and  knowledge  of  successful  online  teachers  in  higher  education.  Cooper  (1998)  states  that  research 
synthes  es  emphasize  “ . . .empirical  studies  and  seekfs]  to  summarize  past  research  by  drawing  overall 
conclusions  from  many  separate  investigations  that  address  related  or  identical  hypotheses  ” (p.3).  Compared  to 
the  meta-analysis  method  which  is  commonly  used  to  synthesize  research  findings,  Cooper’s  method  focuses 
more  on  the  similarities  and  differences  across  research  studies  to  enhance  shared  concepts  and  terminologies  as 
well  as  to  facilitate  the  transition  from  research  to  practice  (Cooper,  1998;  Hare  & Noblit,  1983). 

Coding  sheets  were  generated  to  organize  and  compare  research  findings  according  to  the  following 
guidelines  posited  by  Cooper  (1998).  The  coding  sheet  should: 

1.  Include  detailed  information  about  each  study  that  is  related  to  the  topic. 

2.  Include  seven  categories  of  information  (i.e.,  report  identification,  study  setting,  subjects, 
methodology,  treatment  features,  statistic  outcomes  or  effect  size,  and  the  coding  procedures). 

3.  Be  standardized  to  contain  the  main  comparison  of  study  interests. 

4.  Provide  space  for  descriptive  note  for  each  study  in  order  to  find  the  interaction  between  main 
effect  and  other  variables. 

5.  Be  pilot  tested  and  revis  ed  to  precisely  define  necessary  categories  and  unveil  further 
ambiguities. 

This  study  used  the  studies  by  Thach  (1994)  and  Williams’s  (2003)  to  form  initial  coding  columns  (essential 
skills  and  outputs),  then  modified  columns  by  using  the  other  eight  randomized  studies.  Since  the  topic  of 
distance  education  skills  can  be  viewed  from  different  perspectives,  researchers  used  the  following  criteria  to 
select  research  for  inclusion  in  the  study: 

1.  Research  studies  should  cover  overall  skills  or  outputs  which  educators  can  teach  in  online 
environment  successfully. 

2.  Research  studies  should  mainly  focus  on  higher  education. 

3.  Research  studies  should  mainly  be  conducted  in  the  English  speaking  countries . 

4.  Findings  should  be  proved  by  half  of  the  studies. 

A total  of  twenty  studies  were  used  to  create  the  coding  sheet  (see  Appendix  A for  coding  references), 
including  thirteen  empirical  studies  and  seven  position  papeis  published  between  1994-2004.  This  study 
focused  on  skills  and  knowledge  mainly  found  in  the  higher  education  among  English  speaking  countries. 
Therefore,  the  results  may  not  represent  findings  from  other  areas.  In  addition,  some  studies  do  not  specify  the 
type  of  distance  learning  they  conducted.  Therefore,  those  findings  may  also  be  affected. 

Findings 

What  are  essential  skills  of  a successful  online  distance  educator  in  higher  education? 

Based  on  the  synthesized  results,  there  are  six  essential  skills  and  sixteen  outputs  for  performing  those 

skills : 

1.  Interaction 

■ Guide  and  maintain  interactive  discussion 

■ Provide  timely  feedback 

■ Encourage  peer  learning 

■ Advise  and  counsel  students 

2.  Management 

■ Monitor  and  evaluate  student  performance 

■ Facilitate  presentation 

■ Introduce  support  services  to  students 

3.  Organization/  instructional  design 

■ Provide  clear  learning  outcomes,  objectives,  and  expectation 

■ Organize  materials  and  activities  clearly  and  well 
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■ Identify  students’  learning  styles/needs 

■ Conduct  instructional  design  effort 

■ Present  materials  and  activities 

■ Provide  a variety  of  learning  activities 

4.  Technology 

■ Utilize  technology  in  a competent  manner 

5.  Content  knowledge 

■ Master  in  content  area 

6.  Teamwork  skills 

■ Collaborate  with  technical/support  skills 

The  skill  areas  are  ranked  from  1 -6  in  order  of  importance.  The  outputs  are  grouped  by  area  first,  and 
then  by  importance.  Not  surprisingly,  the  ability  to  stimulate  and  facilitate  interactions  is  the  most  important 
skill  that  online  distance  educators  should  possess.  Even  though  many  educators  urged  the  competency  of 
technology  in  online  teaching,  this  ranking  also  presents  a trend  that  online  education  is  driven  by  pedagogical 
concerns  instead  of  technological  concerns.  Another  interesting  finding  is  the  relative  importance  of  content 
knowledge.  It  seems  that  in  an  online  environment,  educators’  mastery  in  the  content  area  is  important,  but  their 
ability  organize  and  present  content  information  to  students  is  more  important. 

From  the  review  of  literature,  we  also  found  factors  that  may  affect  the  essential  skills  of  successful 
online  distance  educators  in  higher  education: 

1.  Supporting  system  of  the  institutions  (Berge  & Muilenburg,  2001;  Dzuiban,  Shea,  & Arbaugh, 
2004).  The  better  supporting  system  is,  the  less  essential  skills  those  online  educators  need. 

2.  Delivery  methods  of  the  online  courses . Some  online  courses  also  require  partial  classroom 
attendance;  therefore,  the  essential  skills  and  the  outputs  will  be  varied  (Pyle  & Dziuban,  2001). 

3.  Learning  outcomes  of  online  instruction.  Essential  skills  and  outputs  for  conducting  online 
courses  with  higher  thinking  skills  will  be  different  from  those  with  fundamental  operation  skills 
(Pyle  & Dziuban,  2001;  Southern  association  of  college  and  schools,  2000  Dec.) 

4.  Instructional  approach  and  epistemological  beliefs . Different  instructional  approaches  will  affect 
online  distance  educators’  teaching  strategies,  and  different  teaching  strategies  will  affect  what 
kind  of  skills  they  need  to  possess  in  order  to  teach  online  successfully.  It  also  means  that  what 
educators’  epistemological  beliefs  will  direct  them  to  choose  different  instructional  approaches. 
Therefore,  the  skills  or  knowledge  for  teaching  online  successfully  might  be  different  (Gagne, 
Wager,  Golas,  & Keller,  2005;  P.  L.  Smith  & Ragan,  1999). 

Is  there  difference  between  teaching  in  face-to-face  and  online  environment? 

To  examine  the  difference  between  teaching  online  versus  face-to-face  classroom  settings,  we 
identified  key  competencies  specified  for  conventional  classroom  educators  and  reviewed  literature  discussing 
the  paradigm  shift  from  traditional  face-to-face  classroom  to  online  settings.  Like  others,  we  found  that  most 
skills  are  similar  (c.f.,  Coppola  et  ah,  2002;  Easton,  2003;  Palloff  & Pratt,  1999).  However,  a number  of  studies 
also  stated  that  skills,  like  content  and  activities’  organization,  interaction  and  communication,  evaluation  and 
assessment,  office  hour  maintenance,  and  teamwork,  need  to  be  adapted  to  meet  the  requirements  of  an  online 
environment  (Berliner,  1988;  Dzuiban  et  ah,  2004;  Easton,  2003;  SACS,  2000).  For  example,  in  conventional 
classroom  settings,  the  primary  role  of  educators  is  to  instruct,  but  in  the  online  environment,  the  role  changes 
to  an  instructor  and  instructional  designer.  Another  shift  has  the  educator’s  presentation  style  changing  from  a 
lecture  orientation  to  a Socratic  approach.  It  is  important  to  note  that  many  of  the  recommended  adaptations  for 
teaching  online  are  also  being  adopted  for  teaching  in  conventional  face-to-face  classroom  settings  even  though 
they  might  be  less  emphasized. 

Even  though  many  of  the  skills  are  similar,  research  suggests  that  educators  need  training,  support,  and 
faculty  development  to  make  the  transition  from  teaching  in  conventional  classrooms  to  teaching  online..  The 
training  and  support  may  or  may  not  be  housed  in  the  educators’  home  institution.  If  an  institution  asks 
educators  to  teach  online,  but  does  not  provide  adequate  training  and  support,  the  quality  of  online  course 
materials  and  delivery  may  be  jeopardized.  Moreover,  the  institution  may  frustrate  their  educators  without 
assisting  them  through  the  transition 

Is  a self-assessment  available  for  educators  to  determine  their  capability  and  suitability  for  teaching 
online? 
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Several  self-assessment  instruments  have  been  reported  for  determining  whether  educators  in  higher 
education  are  suitable  or  capable  of  teaching  online.  Some  of  them  have  an  online  version  for  educators  to  test 
by  themselves.  Reported  self-assessment  instruments  include: 

1.  Are  you  ready  to  work  online?  (Salmon,  2000) 

2.  Is  online  teaching  right  for  me?  (University  of  Central  Forlida,  2003) 

3.  Is  Online  Teaching  for  Me?  Self-evaluation  Quiz.  (Onlinelearning.net,  n.d.) 

4.  Personality  traits  and  teaching  style  preferences  for  online  instructors  (Fuller,  Norby,  Pearce,  & 
Strand,  2000) 

5.  Self-assessment  of  Readiness  for  creating  or  teaching  an  online  course  (Gummess,  n.d.) 

6.  Teaching  Styles  and  Web  Pages  (Indiana  State  University,  2004) 

What  elements  in  existing  self-assessment  instruments  were  used  in  higher  education? 

Nitko  (2004)  points  out  that  an  authentic  performance  assessment  should  include  two  essential 
elements:  (a)  a performance  task  which  learners  can  demonstrate  by  producing  an  extended  written  or  spoken 
answer,  and  (b)  a clear  rubric  or  criteria  for  grading  that  performance.  By  examining  the  elements  of  the 
existing  instruments,  we  found  that  most  focus  on  specific  perspectives,  such  as  technical  (necessary  computer 
skills),  psychological  (personality  traits)  and  pedagogical  (teaching  style).  Even  though  some  of  the  instruments 
provide  other  elements  which  are  included  in  our  list  of  essential  skills  and  outputs  (e.g.,  “Is  online  teaching 
right  for  me?”  or  “Is  online  teaching  for  me?”)  such  elements  do  not  appear  to  be  grounded  in  learning  theories 
or  supported  by  empirical  studies.  In  addition,  we  did  not  find  evidence  for  validity,  reliability  or  follow  up 
studies  reported  on  most  instruments.  The  review  of  literature  did  reveal  several  guidelines  for  generating  self- 
assessments  : 

1.  The  instruments  should  be  aligned  with  available  training  and  support . For  example,  the  faculty 
support  center  at  the  University  of  Central  Florida  provides  several  consultations  for  faculty 
members  to  assess  their  skills  followed  by  training  and  support  services  to  address  identified 
areas  of  need  (Dziuban  et  ah,  2003). 

2.  Assessment  items  need  to  be  aligned  to  learning  objectives  and  cover  cognitive  (Bloom, 
Englehart,  Furst,  Hill,  & Krathwohl,  1956)  and  affective  (Krathwohl,  Bloom,  & Masia,  1964) 
domains. 

3.  The  instruments  should  be  treated  as  one  sources  of  assessment  data.  Nitko  (2004)  states  that 
assessments  should  be  conducted  from  multiple  perspectives  to  reduce  potential  bias. 

Is  there  an  alternative  way  to  determine  whether  an  educator  in  higher  education  is  suitable  for  teaching 
online? 

Our  findings  show  that  there  are  some  alternative  ways  to  determine  whether  educators  are  suitable 
and  capable  of  teaching  online: 

1.  Be  evaluated  by  professional  organizations:  Some  creditable  organizations  provide  course  or 
certificate  evaluation  services.  For  example,  the  National  Board  for  Professional  Teaching 
Standards  (NBPTS)  provides  many  services  to  assure  the  quality  of  education.  Educators  can 
submit  their  course  materials,  students  ’ works,  even  videos  of  their  class  activities  to  NBPTS  to 
get  feedback  for  their  teaching  performance  as  well  as  their  course  content.  This  kind  of  service 
is  not  mandatory,  and  they  evaluate  the  course  as  well  as  the  certificate  by  using  multiple 
resources . The  advantage  is  that  the  evaluation  result  is  relatively  creditable,  coming  from  an 
independent  source.  On  the  other  hand,  the  process  takes  considerable  time  and  effort  from  both 
ends  and  may  not  be  practicable  as  standard  practice,  particularly  for  larger  organizations. 

2.  Participate  in  assessment  centers . Compared  to  the  previous  option,  this  is  also  a formal  way  to 
access  educators’  capability  of  teaching  online.  However,  the  assessment  materials  are  from  the 
center,  not  from  educators  themselves.  Moreover,  educators  may  need  to  go  to  performance 
assessment  centers  to  complete  simulated  teaching  activities  like  inferences,  peer  collaboration, 
or  staff  development.  Therefore,  this  kind  of  performance  assessment  is  creditable,  but  again, 
may  not  be  practicable  as  standard  practice. 

3.  Follow  checklists,  best  practices,  and  benchmarks  from  creditable  organizations.  There  are  many 
checklists,  best  practices,  and  benchmarks  from  scholarly  journals,  books,  or  organizations 
developed  by  experienced  online  educators  or  experts.  Even  though  those  materials  may  not  have 
been  developed  for  your  exact  environment,  they  may  still  provide  useful  guidelines  for  general 
assessment  purposes. 
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4.  Have  a good  mentor  in  the  same  field.  It  is  always  a good  idea  to  have  a good  mentor  in  the  same 
field  (Perreault,  Waldman,  Alexander,  & Zhao,  2002).  He  or  she  can  provide  suggestions  which 
match  the  environment  you  have,  even  though  sometime  his  or  her  suggestions  may  be 
somewhat  subjective.  . 

Recommendations 

Based  on  our  findings,  we  recommend  that: 

1.  Researchers  conduct  additional  studies  to  determine  how  specific  disciplines,  delivery  modes, 
and  learning  outcomes/strategies  affect  the  skills  and  knowledge  necessary  for  distance  educators 
to  teach  successfully  online.  Even  though  some  speculate  that  skills  and  knowledge  vary  by 
domain,  course  delivery  mode,  and  desired  learning  outcomes , there  is  little  empirical  evidence 
to  support  such  claims . 

2.  Researcher  and  partitions  determine  and  report  the  validity  and  reliability  of  self-assessment 
instruments.  Even  though  we  found  many  self-assessments  to  help  distance  educators  determine 
if  they  are  prepared  and/or  suitable  to  teach  online,  the  validity  and  reliability  of  such 
instruments  are  rarely  reported.  Additional  follow  up  studies  are  needed  for  the  self-assessments. 
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Indiana  University 

Introduction 

The  term  “customized  training”  has  recently  been  used  often  when  discussing  training.  Many  training 
development  companies  and  community  colleges  selling  their  training  programs  use  it  frequently  for  engaging 
prospective  customers  and  companies  considering  purchase  of  their  products.  So,  what  is  customized  training? 

Definition  of  Customized  Training 

A shift  has  emerged  in  corporate  training1  from  a training-driven  approach  to  a human  performance- 
driven  approach.  In  the  other  words,  people  want  the  training  to  impact  the  bottom  line  by  enhancing 
performance,  not  just  for  the  training  to  occur  for  the  sake  of  training  itself.  In  order  for  training  to  influence 
business  significantly,  it  needs  to  be  closely  aligned  with  business  goals  and  also  be  designed  to  address  these 
goals  from  the  beginning.  To  fulfill  corporate  leaders’  growing  need  for  training,  it  is  critical  that  training 
should  be  customized  for  a specific  company  and  for  specific  target  audience  and  customization  be  embedded  in 
the  design  and  development  of  training  products.  So  what  do  we  mean  by  customization? 

According  to  the  definition  in  a dictionary,  the  term  “customize”  is  to  make  or  change  something  to 
suit  the  needs  of  the  owner  (Oxford  Advanced  Learner’s  Dictionary,  6th  Ed.).  By  employing  this  definition, 
customized  training  can  be  defined  as  training  made  or  changed  to  address  the  needs  of  the  target  learners. 

One  training  development  company  website  defines  customized  training  as  follows: 

“Customized  training  is  the  process  of  tailoring  training  to  organization’s  needs  either  by  adapting  an 

existing  program  or  developing  a new  program. . . .We  can  customize  one  of  our  existing  programs  or 

develop  a course  for  you  from  the  ground  up. 

(http://www.corexcel.com/html/customized.training.contact.hours.htm)” 

In  addition.  New  York  State  Department  of  Labor  defines  customized  training  as  training  “that  is 
designed  to  meet  the  special  requirements  of  an  employer  or  group  of  employers . 
(http://www.workforcenewyork.org/qaojtcustrng.html)” 

In  the  two  definitions,  one  from  a training  development  company  and  the  other  from  a department  of 
labor,  customization  requirements  of  employers  and  interests  of  organizations  is  emphasized  since  they  regard 
customized  training  as  products  in  high  demand.  In  these  definitions,  both  consideration  for  employees  and  the 

emphasis  on  the  end  user’s  needs  are  invisible . Blackmon  and  Rehak  (2003)  put  much  more  weight  on  trainees 

2 

than  employers  in  the  definition  of  customized  training." 

They  described  a model  for  role  and  competency-based  customization  by  using  learning  technology 
standards  (e.g.  learner  profiles,  competency  definitions,  sequencing  rules,  learning  objects).  One  of  the  most 
salient  characteristics  of  the  model  lies  in  dynamic  assembly  of  content  objects  extracted  from  content 
repositories  to  enable  the  creation  of  customized  learning,  and  dynamic  creation  of  optimal  learning  strategy 
that  suits  the  needs  of  the  learners.  We  can  learn  several  ways  to  customize  learning  by  considering  roles  and 
competencies  of  the  target  learners  while  customizing  content  and  learning  activities. 

The  characteristics  described  in  the  definitions  presented  so  far  are  not  sufficient  to  grasp  a whole 
picture  of  customized  training. 

Purpose  of  Study 

Throughout  all  these  definitions  (both  of  training  and  customized  training),  it  is  very  clear  that  there 
are  two  different  perspectives  on  customized  training:  employer’s  vs.  employee’s  standpoint.  However,  in  this 
study,  these  bipolar  standpoints  may  not  be  differentiated  under  the  assunption  that  all  the  training  is  intended 
and  designed  to  impact  the  bottom  line  in  the  long  run. 

The  purpose  of  this  study  is  to  explore  the  building  blocks  of  the  concept  of  “customized  training” 
beyond  its  conventional  definitions  and  to  form  a comprehensive  conceptual  framework3.  Furthermore,  the 
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current  practice  of  designing  and  developing  customized  training  in  corporations  is  explored  and  investigated  to 
answer  the  following  research  questions,  which  are  two -fold.  For  the  product  dimension  of  customized  training, 
reasons  and  ways  for  customizing  training  will  be  addressed.  For  the  process  dimension,  major  issues  faced  and 
activities  involved  in  the  design  process  of  customized  training  will  be  identified. 

For  the  product  dimension,  the  four  questions  will  be  addressed: 

• There  must  be  assumptions  about  training.  These  can  serve  as  rationale  for  customization.  Why  is 
training  customized? 

• The  objects  that  can  be  customized  may  vary  from  content  to  methods  to  resources.  What  is  customized 
in  training? 

• Training  can  be  customized  with  regards  to  individual  role  and  competency  or  current  level  of  skill  and 
knowledge.  Sometimes,  individual  attitude  or  learning  style  can  be  considered  for  customization.  What 
do  they  consider  for  customization  of  training? 

• In  addition,  how  can  the  current  customized  training  be  improved  for  better  quality? 

Description  of  the  current  process  of  design  and  development  for  customized  training  can  give 
practical  guidelines  for  practitioners.  It  is  possible  that  any  specialized  process  of  customized  training  exists. 

For  the  process  dimension,  three  questions  will  be  addressed. 

• What  are  the  major  elements  of  design  process  for  customized  training? 

• What  are  the  major  issues  instructional  designers  have  faced  in  the  process  of  design  for  customized 
training? 

• What  is  the  ideal  process  of  designing  customized  training  like?  How  should  your  current  design 
practice  be  improved? 

Answers  to  the  questions  will  provide  building  blocks  that  can  describe  the  current  practice  and 
systems  for  customized  training.  Understanding  these  components  and  their  relationships  can  give  insight  to 
training  designers  on  how  to  approach  and  actually  develop  customized  training.  Design  and  development  for 
customized  training  needs  a conceptual  framework  which  can  include  all  relevant  elements  and  can  depict 
relationships  among  those  elements.  The  conceptual  framework  will  help  training  designers  to  understand 
elements  to  be  considered  in  training  design  and  to  see  those  elements  from  a systemic  point  of  view.  The  value 
of  this  study  may  lie  in  explicit  visualization  of  building  blocks  of  design  and  development  of  customized 
training  residing  tacitly  in  training  designers’  mind. 


Method 

Research  Design 

Due  to  deficient  amounts  of  corporate  research  on  customized  training,  it  is  necessary  to  meet 
practitioners  who  have  experience  in  designing  and  developing  customized  training  and  to  hear  their  narratives 
in  order  to  understand  the  current  practice  and  to  investigate  the  research  questions. 

As  Yin(2002)  reported,  “the  case  study  has  peculiar  advantages  when  research  questions  are  related  to 
how  and  why,  when  the  investigator  has  little  control  over  events,  and  when  the  focus  is  on  a contemporary 
phenomenon  within  some  real-life  context  (p.  1 ).”  In  order  to  utilize  the  full  advantage  of  the  case  study, 
multiple  methods  were  integrated.  Hence,  seven  face-to  face  interviews  as  well  as  rich  and  in-depth  document 
analysis  of  web  sites  and  documents  acquired  from  the  interviewees  were  employed.  When  necessary,  a follow- 
up interview  was  conducted  via  email  or  on  an  online  synchronous  chatting  tool. 

Cases 

The  units  of  analysis  were  seven  training  designers  (developers)  who  have  experiences  in  customized 
training.  Real  names  of  the  interviewees  are  replaced  by  pseudonyms. 

Kevin  has  10  years  experience  as  a training  developer  and  is  working  as  vice  president  in  a custom 
training  development  company  located  in  the  mid-western  part  of  the  United  States.  Michael  has  2 years 
experience  as  a training  designer  and  is  working  for  the  same  company  as  Kevin.  Stella  is  working  as  a task 
analyst  in  a pharmaceutical  company  which  is  also  in  the  mid  west.  Brad  has  5 years  experience  as  a course 
developer  and  is  working  for  the  same  company  as  Stella.  Eugene  has  2 years  experience  as  a training  designer, 
working  for  a family  restaurant  company  in  Korea.  Mira  is  working  as  a training  designer  in  a company  that 
produces  food  and  animal  feed  additives  in  Korea.  Susan  is  a training  designer  in  a company  providing 
information  technology  s ervices.  Kevin  and  Michael  are  working  in  a company  that  develops  customized 
training  for  the  client  companies.  The  remainders  are  developing  in-house  training  programs . In  summary,  three 
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male  and  four  female,  ages  ranging  from  twenties  to  thirties  were  interviewed. 

Data  Collection 

The  Interview  protocol  was  developed  by  adapting  research  questions  for  a more  communicative  style . 
The  protocol  was  pilot-tested  with  peers  to  identify  unforeseen  problems  and  was  revised  based  on  the  results  of 
the  pilot  test.  For  face  validity  of  the  instrument,  two  experts  reviewed  the  protocol,  which  was  then  modified 
according  to  their  comments.  All  questions  were  open  to  allow  new  or  modified  questions  as  the  interviews 
progressed. 

Four  interviewees  in  the  U.S.  were  interviewed  face-to-face.  Three  of  the  interviews  took 
approximately  60  minutes  while  the  one  with  Brad  took  30  minutes.  All  the  interviews  were  recorded  with  a 
digital  voice  recorder  after  acquiring  permission  and  later  transcribed  word  by  word.  The  interviews  with  the 
other  three  in  Korea  were  conducted  through  an  instant  messaging  tool.  All  the  texts  of  the  interviews  were 
saved  as  text  files  for  further  analysis.  For  validity,  the  researcher  explained  the  scope  and  the  purpose  of  this 
research  to  the  interviewees  to  make  sure  that  both  sides  interpreted  the  research  questions  in  the  same  way. 

Based  on  data  from  the  first  interview,  the  follow-up  questions  were  sent  via  email  to  the  interviewees 
in  cases  where  further  elaboration  or  clarification  was  needed.  Through  the  follow-up  interviews,  additional  in- 
depth  and  data  were  gathered. 

During  the  interview,  the  researcher  took  notes  that  were  utilized  for  asking  more  in-depth 
questions  and  for  clarifying  interviewee’s  answers. 

Data  Analysis 

After  completing  data  collection,  the  briefly  summarized  extracts  of  the  interviews  were  sent  to  the 
interviewees  to  ensure  reliability  as  a means  of  member  checking.  The  transcripts  were  reviewed  several  times 
in  order  to  find  emergent  themes. 

Extensive  literature  was  reviewed  for  gathering  data  to  establish  the  conceptual  framework  of 
customized  training.  Considerable  amount  of  web  documents  related  to  customized  training  were  collected 
through  the  Google  search  engine  and  were  later  analyzed.  Additionally,  a recorded  presentation  by  Kevin  and 
Michael,  a corresponding  handout,  and  a whitepaper  of  their  company  were  analyzed. 

Results 

Why  is  training  customized? 

Training  should  impact  the  bottom  line  From  the  organization’s  perspective,  training  should  be  a 
means  for  increasing  profit.  In  order  for  training  to  address  this  need,  it  is  required  to  be  tightly  aligned  with 
business  goals.  As  a rule,  HRD  strategies  are  established  in  conjunction  with  business  strategies.  Therefore, 
training,  one  of  interventions  driven  by  the  strategies,  is  also  expected  to  play  a role  in  accomplishing  business 
goals.  Without  considering  business  outcomes  or  having  clear  output  images  of  training,  training  might  be 
implemented  in  vain.  In  this  sense,  training  should  be  customized  to  meet  defined  corporate  business  goals. 

“I  think  all  training  in  my  company  is  customized.  From  the  company’s  perspective,  training  is 

used  as  one  strategy  that  helps  employees  accomplish  short-term  and  long-term  goals  of  the 

company.  Therefore,  unless  it  is  not  optimized,  people  complain  about  the  training.  Unlike 

schools,  corporations  have  very  concrete  objectives  and  targets  and  require  visible  outcomes.  We  try 
to  avoid  unnecessary  training  that  does  not  lead  to  business  results.  Therefore,  business  goals,  business 
strategies  and  HRD  strategies  that  inherited  from  the  top  management  should  be  aligned  with  all  training 
(Eugene).” 

Business  goals  keep  changing  according  to  market  trend,  emergence  of  new  product  lines,  change  of 
management  lines  and  environmental  disruption.  Training  should  be  very  keen  to  those  changes  and  should 
respond  to  them  aggressively.  Most  of  the  interviewees  agreed  that  if  it  is  separated  from  ever-changing 
business  goals,  training  cannot  satisfy  the  company  or  target  learners. 

The  characteristics  of  the  target  audience  are  very  diverse  Just  like  other  higher  education  fields, 
corporate  training  is  very  likely  to  encounter  diversity  in  the  target  audience  with  regards  to  skill  and  knowledge 
level,  learning  style,  and  prior  experience  to  name  just  a few.  Especially  if  the  company  has  several  sub- 
industries and  needs  a training  program  to  teach  all  employees  across  the  industries,  customization  is  crucial  by 
taking  into  account  the  target  learners’  diverse  characteristics,  contexts  where  the  training  would  take  place,  and 
the  way  that  trainees  would  apply  what  they  learn  to  their  jobs. 
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We  have  our  own  way  Every  organization  has  its  own  procedure  and  learning  culture.  Of  course  all 
training  cannot  be  developed  by  the  organization  itself.  A considerably  large  proportion  of  training  is  supplied 
by  vendors  and  neighbor  universities.  However,  those  training  programs  produced  by  others  may  not  address  all 
organization’s  unique  procedure  suitably  nor  care  about  the  contexts.  Therefore,  training  designers  who  build  up 
in-house  training  programs  feel  that  they  need  to  provide  customized  training  that  fully  attend  to  their  own 
training  needs  and  touch  on  their  own  unique  way  of  doing  things,  especially  if  there  is  no  adequate  training 
material  on  the  market. 

“We  customize  because  no  materials  exist  either  internally  or  on  the  market  place.  In  our  field 
there  isn't  a lot  of  pre  -built  training  on  sophisticated  lab  techniques  or  procedures.  Also,  ‘the 
[pharmaceutical  company  name  ] way’  is  usually  a little  more  stringent  than  standard  practice 
because  of  FDA  guidelines.  So,  a university  course  on  how  to  perform  a technique  wouldn't  be 
adequate  (Brad).” 

“The  sort  of  business  needs  that  they’re  addressing  for  companies  or  organizations.  They  cannot  be 
adequately  fit  by  a standardized  generic  training  or  books. . . often  times  their  organizational  problems 
that  are  very  specific  to  [their]  organization. . .you  have  your  own  culture  and  your  own  systems  and  so  you 
need  someone  who  can  learn  about  and  understand  your  specific  challenges 

and  then  create  training  that  addresses  those  specific  challenges  and  in  your  context  (Michael).” 

If  not  customized,  training  doesn 't  work  Compared  to  training  designers  for  in-house  programs, 
training  program  development  companies  might  be  more  sensitive  to  the  degree  of  customization  of  the 
programs  they  sell  to  the  client  companies  because  training  in  business  settings  never  takes  place  in  a vacuum 
(Molenda  & Pershing,  2004,  p.26)  and  the  context  where  training  would  occur  is  hard  to  grasp  at  a glance  from 
outsiders.  Kevin  and  Michael  stressed  consideration  of  contexts  where  a specific  company  is  located  when  they 
design  and  develop  customized  training.  Also,  they  believed  that  there  should  be  diverse  versions  of  training 
customized  to  different  target  learners.  They  asserted  that  any  training  that  does  not  consider  the  contexts  is 
likely  to  fail.  They  reported  that  they  need  to  understand  what  the  target  learners’  jobs  are  and  how  they  apply 
what  they  learn.  Kevin  stressed  that  training  is  not  training  at  all  but  just  information  if  it  does  not  consider  the 
context.  Training  designers  should  tailor  training  programs  to  address  the  context. 

“As  a matter  of  fact,  it  doesn’t  work  if  you  don’t  customize  it.  I think  so  much  instruction  doesn’t 
work.  They  try  to  address  too  broad  audience  and  to  fit  in  many  organizations...  You  can  write  one 
book  about  management.  But  too  many  times,  people  don’t  learn  from  that  book.  You  have  to  create  an 
opportunity  to  learn  from  the  book.  You  really  want  me  to  apply  management  back  in  my  job.  You 
need  to  understand  what  my  job  is.  Need  to  understand  how  I apply.  That’s  the  principle.  (Kevin)” 

“I  don’t  think  that’s  instructional  design  inherently  unless  it  focuses  on  customization.  That’s  the 
design  of  our  instructional  design.  Without  it,  it’s  just  information.  It’s  the  instruction  that  doesn’t  take 
learners,  context  and  business  into  account  (Kevin).” 

What  is  customize  din  training? 

Based  on  data  collected  from  the  in-depth  interviews  with  the  seven  training  designers  and  document 
analysis,  a taxonomy  for  what  to  customize  in  training  is  built  in  terms  of  what  to  learn,  how  to  learn,  when  to 
learn  and  where  to  learn.  By  incorporating  all  the  results,  a provisional  conceptual  framework  (see  Figure  1) 
was  constructed. 

What  to  learn  All  the  interviewees  reported  that  they  customize  content  to  specific  target  learners. 

They  adapt  the  scope  and  the  depth  of  content.  Sometimes  they  reorganize  the  content  by  adding  real  cases  that 
help  learners’  understanding.  Based  on  the  result  of  front-end  analysis,  they  customize  the  scope  and  the  depth 
of  content  by  taking  the  level  of  current  knowledge  and  skill  related  to  the  content  into  account. 

When  there  is  a broad  range  of  target  learners,  they  create  multiple  versions  of  the  same  course  by 
customizing  the  content. 

“If  we  have  a wide  audience  or  a split  audience  with  different  needs  and  skill  levels,  we  consider  if 
multiple  versions  of  a course  would  be  the  best  solution.  Recently  we  had  a procedure  that 
everyone  in  my  department  was  required  to  be  trained  on.  However,  non-chemists  didn't  require  as 
deep  an  understanding  of  the  details  of  the  procedure.  So,  we  tailored  the  learning  to  the  audience's 
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skill  level  and  made  the  non-chemist  course  a little  more  basic  with  different  learning  outcomes  and 
more  background  material.  When  appropriate  we  used  course  materials  from  the  chemist  course, 
which  was  more  in-depth  (Brad).” 

So  far  the  designers  have  reported  that  they  customize  what  is  to  be  learned  by  target  learners  during 
the  design  and  development  phase.  However,  there  is  another  phase  where  customization  frequently  takes  place. 
Training  designers  and  supervisors  of  employees  arrange  and  assign  training  programs  to  specific  persons  by 
considering  their  responsibilities  and  jobs.  This  phase  comes  after  all  training  materials  are  already  developed. 
“Everyone  has  individual  training  plan.  The  training  plan  lists  the  classes  and  courses  they  need  to 
take.  Matching  classes  with  job  category  is  one  of  my  tasks.  This  is  considered  customization.  We 
have  the  courses  like  pieces,  now  we  place  those  pieces  to  address  job  category  needs.  The  object  of 
customization  is  primarily  about  the  content  (Stella).” 

“We  defined  general  competencies  that  our  company  expects  all  employees  to  have.  But  they  are  not 
job  or  task-  specific  competencies.  Anyway,  all  employees  can  access  a tool  that  diagnoses 
their  own  competencies.  According  to  the  results,  the  tool  would  prescribe  training  solutions 
including  other  resources  for  enhancing  their  competencies  to  the  specific  learner  (Eugene).” 

How  to  learn  Once  the  topic,  the  scope  and  the  depth  of  the  content  are  determined,  the  designers 
customize  instructional  strategies  or  learning  activities  related  to  “how  to  learn”  along  with  the  selected  content. 

“In  order  to  get  performance,  you  want  to  focus  on  wonderful  performance,  then  you  have  to 
develop,  you  can  keep  content  system.  It’s  the  context  that  you  are  to  customize.  How  am  I going  to 
apply  this  information?  Not  what  is  the  content?  So  I talked  to  my  clients.  You  keep  the  basic 
management  content.  But  what  you  need  to  change  is  the  context  in  which  I apply.  (Kevin)” 

Customization  of  the  learning  process  involves  considerations  on  the  sorts  of  content  and  accessibility 
of  target  learners  to  the  training  such  as  infrastructure,  time  availability  and  their  understanding  of  instructional 
strategies  and  learning  activities. 

“Also,  we  customize  mode  of  delivery.  By  taking  into  account  the  characteristics  of  individual 
task,  we  select  the  most  effective  mode  of  delivery.  For  example,  there  is  a topic  that  cannot  be 
trained  through  off-the-job  training.  Ultimately,  the  reason  why  we  optimize  the  mode  of  delivery  with 
the  training  content  and  objective  is  to  maximize  the  effect  of  the  training.  Also  we  customize 
learning  activities  and  other  instructional  strategies  (Eugene).” 

When  developing  a training  program  for  all  employees  across  the  departments  of  a company,  training 
designers  consider  characteristics  of  industries  because  certain  employees  in  a specific  industry  have  preference 
to  specific  instructional  strategies.  They  also  customize  performance  support  strategies  including  EPSS 
(Electronic  Performance  Support  System) , job-aids,  and  informal  mentorship  programs  according  to  the 
characteristics  of  training  topics . 

When  to  learn  This  facet  of  customization  depends  heavily  on  organizational  learning  culture  and 
availability  of  employees  to  be  trained.  If  your  organization  encourages  self-regulated  learning,  overall  training 
would  be  provided  with  a form  of  printed-self-study  materials,  job-aids,  EPSS,  and  web-based  instruction.  In 
order  to  allow  flexibility,  the  training  designers  select  appropriate  mode  of  delivery. 

They  also  customize  the  timing  of  training  in  order  to  assure  effectiveness  and  transferability  of  what 
they  learn  from  the  training. 

“It  is  not  reasonable  to  train  people  on  something  that  they  are  going  to  perform  in  6 months  from 
now.  It  does  not  make  any  sense.  You  will  forget  everything.  I think  that  we  are  making  a change  on 
their  individual  training  plans  that  will  address  their  needs.  Instead  of  assigning  them  training  that  is 
not  relevant  right  now,  we  just  assign  the  training  when  they  are  close  to  performing  the  task.  We  are 
calling  it  just-in-time  training.  When  a maintenance  guy  receives  a work  order  (request  to  fit 
something),  he  or  she  will  request  training  on  the  task  that  the  work  order  requires.  He  or  she  will  take 
the  training  right  before  to  performing  the  task  so  that  he  or  she  does  not  forget  the  information 
(Stella).” 
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Timing  is  one  of  the  aspects  that  training  designers  invariably  customize  in  their  practice. 


Where  to  learn  There  are  several  options  on  where  to  learn.  In  previous  years,  training  usually  took 
place  in  an  instructor-led  classroom  However,  the  places  for  training  can  be  customized  by  taking  into  account 
sorts  of  topic,  time -availability  and  effectiveness  of  training.  In  order  to  avoid  unnecessary  discontinuity  of  jobs 
and  to  reduce  expenditure  on  travel  costs  required  to  send  employees  out  of  the  workplace,  on-the-job  training, 
printed-self-paced  materials,  and  corporate  universities  are  options  for  customizing  the  location  of  the  training. 

What  is  considered  for  customization  in  training? 

The  items  answered  to  this  question  were  interpreted  as  factors  that  training  designers  focus  on  during 
the  analysis  phase  as  well  as  align  with  training  programs.  The  quoted  factors  by  the  interviewees  must  play  an 
important  role  in  current  practice  of  design  of  customized  training. 

Business  needs  Business  needs  determine  the  direction  of  overall  HRD  strategies.  Training  is  one  of 
the  interventions  to  increase  employees’  performance  and  to  reduce  performance  gap  between  the  current  status 
and  the  desired.  Performance  is  expected  to  be  evaluated  with  measurements  of  tangible  contributions  to 
business  outcomes.  Therefore,  training  designers  should  assure  that  their  products  are  addressing  ever-changing 
business  needs  by  aligning  training  programs  with  such  needs . All  participants  reported  business  needs  as  the 
most  important  aspect  they  tried  to  align  with  training.  These  business  requirements  drive  training  development 
more  strongly  than  any  other  needs.  They  take  precedence  over  individual  learner’s  needs  or  department’s 
needs. 


Organizational  learning  culture  -Organizations  have  their  own  unique  culture  s in  many  regards.  The 
company  where  Eugene  works  has  stressed  self-regulated  learning  culture  in  the  last  few  years.  Therefore,  the 
training  designers  have  designed  and  developed  training  to  leverage  its  learning  culture.  They  focus  more  on  the 
flexibility  of  timing  and  location  of  training  to  target  learners.  They  also  developed  self-paced  materials  rather 
than  instructor-led  in-class  training  programs.  This  is  one  example  of  customization  of  mode  of  delivery  of 
training  to  fit  organizational  culture. 

Collective  learner  characteristics  All  interviewees  invariably  reported  that  they  align  the 
characteristics4  of  the  target  audience  with  training  programs. 

Other  participants  also  reported  that  they  analyze  the  comprehension  of  instructional  strategies  or 
learning  activities,  educational  background,  and  training  preference. 

Individual  learner  profile  In  addition  to  collective  consideration  of  learner  characteristics  at  design 
and  development  phases , training  designers  also  consider  the  individual  learner  profile  including  his  job,  tasks, 
required  competencies,  history  of  training,  and  his  career  plan  during  the  development  phase  of  an  individual 
training  plan. 

“All  employees  can  access  a tool  that  diagnoses  their  own  competencies.  According  to  the  results,  the 

tool  would  prescribe  training  solutions  including  other  resources  for  enhancing  their 

competencies  to  the  specific  learner.  We  also  have  a plan  to  develop  a diagnosis  tool  for  job- 

specific  or  task-specific  competencies  (Eugene).” 

Content  Characteristics  Depending  on  the  given  content  characteristics,  training  designers  try  to  align 
the  topic  with  appropriate  instructional  strategies.  The  characteristics  include  content  domain  (e.g.  affective, 
cognitive,  and  psycho-motor),  complexity,  and  volatility  (e.g.  stable  or  unstable). 

Context  in  which  learners  apply  what  they  learn  In  order  to  transfer  what  is  learned  from  training  into 
job  performance,  training  designers  need  to  understand  contexts  in  which  the  target  learners  apply  what  they 
learn.  Even  with  the  same  skills,  learners  have  different  environments,  different  tasks  to  do,  and  different  ways 
to  apply  what  they  learn.  Therefore,  training  designers  try  to  align  this  context  with  training  programs  by 
thorough  analysis  of  the  job  and  the  environment. 

All  of  the  themes  and  factors  identified  through  interviews  and  document  analysis  were  integrated  into 
a single  conceptual  framework  which  describes  the  concept  of  “customized  training”. 
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How  can  the  current  customized  training  be  improved  for  better  quality? 

Most  participants  shared  the  sense  that  they  are  not  fully  satisfied  with  their  current  customized 
training  and  that  there  is  room  for  improved  customization.  Even  though  they  customize  training  by  considering 
principal  playing  factors  identified  from  needs  analysis,  learner  analysis,  and  environment  analysis,  they  all 
reported  they  are  still  lacking  comprehensive  and  rigorous  analysis  owing  to  time  and  cost  constraints.  They  all 
agreed  that  the  more  time  they  can  spend  on  analysis  and  the  better  customized  product  they  will  be  able  to 
create. 

One  interviewee  reported  that  it  is  very  important  for  him  to  understand  the  client’s  business  and 
learning  objectives.  Also,  accessibility  to  subject  matter  experts  (SMEs)  that  can  provide  what  he  wants  is 
necessary.  Close  communication  between  training  designers  and  SMEs  should  last  until  they  ship  deliverables 
to  the  client. 

“Having  a clear  understanding  of  what  their  business  and  learning  objectives  are,  having  access  to 
SMEs  that  actually  are  experts  and  can  review.  Making  sure  that  they  give  you  content  you  need. 
Having  SMEs  review  your  work  and  give  feedback  in  timely  manner,  because  we  need  to  translate 
them  into  instruction  properly  and  accurately.  That’s  very  important  as  well.  (Brad)” 

What  are  the  major  elements  of  a design  process  especially  for  customized  training? 

In  contrast  to  prior  expectation,  all  the  interviewees  did  not  distinguish  the  design  process  for 
customized  training  from  the  process  for  one -fits -for-all  training  because  they  all  felt  that  what  they  work  on  is 
to  design  and  build  up  customized  training. 

“I  guess  most  training  programs  that  companies  develop  by  themselves  are  customized.  It’s  very 
common  practice  in  the  corporate  sector.  So,  it  sounds  a little  bit  awkward  to  label  our  practice 
especially  as  customization.  Customization  is  being  regarded  as  a must-have  in  all  training  design 
practice  (Mira).” 

All  respondents  followed  Analysis  -Design-Development-Implementation-Evaluation  (ADDIE)  process 
model.  A prominent  point  drawn  from  the  interviews  is  that  they  all  put  substantial  weight  on  the  analysis  stage 
of  customization. 

Michael  stressed  iterative  processes  through  recurrent  and  very  close  communication  between  the 
clients  and  the  designers  from  the  very  beginning  of  the  design  process.  All  of  the  processes  include  the  initial 
design  of  the  company  and  approval  from  the  clients  iteratively. 

“We  followed  ADDIE  model. ..For  new  clients,  to  develop  one  hour  of  online  training,  it  may  take  me 
16  weeks  to  develop  a course.  For  16  weeks,  maybe  4 weeks  are  for  analysis  and  design.  Another  6 
weeks  [are]  for  development.  Another  2-3  weeks  [are]  for  implementation  and  evaluation.  At  the  end 
of  the  analysis  and  design  phase,  you  have  what  you  want  to  have.  In  the  development  phase,  you  have 
prototyping.  During  the  prototyping,  you  will  get  feedback  from  the  target  audience  to  see  if  our 
assumptions  are  true  and  our  design  matches  what  we’re  looking  for.  We’re  not  going  to  be  100%  right 
but  as  we  develop  it  we’re  still  asking  the  same  questions  again.  Is  our  assumption  true?  Is  this  a right 
balance  between  media  and  content?  We  keep  asking  these  questions  until  we  ship  it  to  the  clients 
(Kevin).” 

In  Kevin’s  case,  they  spend  one  quarter  of  the  whole  development  process  in  analysis  and  design. 
Actually,  the  reason  why  he  did  not  separate  analysis  and  design  into  two  seemed  that  two  phases  take  place  at 
the  same  time  and  they  are  very  closely  interwoven.  Also,  he  stressed  that  they  continue  to  show  interim  outputs 
to  the  clients  and  the  target  learners  in  order  to  assure  they  are  on  the  right  track  in  design  and  they  are  meeting 
the  client’s  expectations.  All  feedback  from  the  clients  is  used  as  a formative  evaluation  for  revision. 

Training  designers  who  develop  in-house  training  stated  that  almost  all  training  programs  are 
customized  to  address  specific  business  needs,  organizational  traits  and  target  learners’  characteristics.  They 
mostly  build  up  training  programs  from  the  ground  up.  It  is  very  rare  to  produce  training  by  modifying  or 
adapting  existing  training  programs  to  specific  needs  or  different  target  audience.  The  process  does  not 
significantly  differ  from  what  training  designers  in  training  development  companies  do.  Differences  lie  in  that 
in-house  training  program  developers  only  contacted  internal  customers  in  their  own  organizations.  Before 
entering  the  development  phase,  they  tried  to  make  sure  that  it  makes  sense  to  all  related  stakeholders. 

However,  most  interviewees  did  not  mention  using  a summative  evaluation  to  see  if  the  training  is 
effective  and  lead  the  business  outcomes.  Brad  admitted  that  evaluation  is  the  weakest  point  in  the  whole 
training  production  process. 
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What  are  the  major  issues  in  the  process  of  designing  for  customized  training? 


Getting  help  from  stakeholders  Most  participants  reported  that  they  definitely  need  to  involve 
interested  parties  such  as  management,  target  learners,  decision  makers  and  subject  matter  experts  in  the  whole 
process  of  designing  and  developing  customized  training.  They  asserted  that  it  is  critical  to  get  SMEs, 
management  and  target  learners  to  help  them  identify  true  training  needs  and  to  understand  the  context, 
learners’  characteristics  is  really  critical  to  their  practice. 

However,  stakeholders  reported  that  they  are  so  busy  with  their  own  work  that  they  cannot  spare  the 
time  to  provide  the  necessary  information  to  them  and  to  establish  a long-lasting  relationship  with  HRD  people. 
There  is  a shared  sense  among  developers  that  if  they  acquire  commitment  and  support  from  management  and 
target  learners,  their  work  would  be  much  more  successful  than  it  is  now. 

Design  trade-off  Training  developers  strive  to  find  equilibrium  between  investment  in  customization 
and  its  outcomes.  With  the  limited  time  and  budget,  they  are  asked  to  produce  training  to  maximize  profits. 
“More  customized?  No,  it’s  already  customized. . . I think  there  should  be  limitations  in  our 
process  where  we  make  certain  aspects  of  the  courses  that  are  very  similar  to  each  other.  To  do  a 
completely  different  style  of  course  for  every  client  would  be  less  efficient.  There  is  always  a 
design  trade-off.  (Michael).” 

“Invested  time  and  efforts  for  customizing  training  should  turn  out  to  be  a result  (Eugene).” 

What  training  developers  usually  do  is  to  portray  “persona”,  that  is,  collective  characteristics  of  target 
learners  from  the  learner  analysis  and  to  customize  training  programs  to  the  persona’s  needs  for  the  pursuit  of 
efficiency.  These  days,  however,  there  are  a system  and  technology  standards  (e.g.  Content  Structure  (CP 

2002) ,  Competency  Definitions  (RCD  2002)  Learner  Information  Package  (LIP  2002),  and  Sequencing  (SS 

2003) )  to  prescribe  required  training  content  and  optimal  learning  strategies  to  a specific  individual  on  demand 
with  the  aid  of  technology  (Blackmon  and  Rehak,  2003). 

Don ’t  assume  and  be  ready  to  listen  Customization  always  requires  comprehensive  and  thorough 
analysis  of  learners,  contexts,  organizations,  and  environmental  factors  that  influence  the  effect  of  training. 
Therefore,  most  interviewees  agreed  that  more  thorough  analysis  can  critically  enhance  the  quality  of 
customized  training. 

As  Susan  reported,  having  subject  matter  experts  or  trainers  review  and  confirm  the  identified  needs 
from  their  perspectives  is  very  critical  in  customization.  Verification  from  diverse  perspectives  on  training 
needs  should  be  done  before  stepping  into  the  design  phase. 

“I  learned  that  the  most  important  [thing]  in  customizing  training  is  to  listen.  People  tend  to 
assume  what  others  need  and  want  without  listening  (they  may  ask  though).  This  is  what  has 
historically  been  happening  in  the  Performance  Improvement  group. . .1  learned  that  if  I have  not 
asked  people  what  they  need,  if  I have  not  understood  what  they  do,  if  I have  not  met  people  in 
their  work  environments  (e.g.  buildings,  offices,  manufacturing  areas)...  I would  not  be  able  to 
customize  their  training  because  I would  not  know  what  their  wants,  needs  and  frustrations  are 
(Stella).” 

Most  participants  emphasized  that  making  assumptions  regarding  training  needs,  learners, 
environments,  and  contexts,  should  be  avoided.  They  knew  very  well  that  training  designers  tend  to  start 
working  on  training  design  with  rough  assumptions  which  are  often  not  based  on  real  data  due  to  time 
constraints  and  heavy  workload. 

Furthermore,  all  organizations  keep  changing  in  all  aspects,  including  organizational  hierarchy, 
strategies  for  market,  and  job  description.  Therefore,  training  designers  should  be  very  sensitive  to  all  of  these 
minute  changes  within  the  organization.  If  not,  they  may  overlook  critical  factors  to  consider. 

Discussion 

By  using  semi  -structured  interview  protocol,  the  researcher  improvised  questions  to  clarify  or  to  ask 
more  about  unexpected  answers  from  the  participants.  Here,  the  things  that  deserve  attention  are  discussed. 
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Design  from  scratch  & low  rate  of  reuse  of  existing  training  All  the  participants  reported  that  they 
usually  design  from  scratch  and  it  is  rare  to  reuse  or  adapt  existing  learning  objects . Even  though  they  said  they 
are  using  learning  management  systems  in  their  companies  that  are  compliant  to  learning  objects  standard  (e.g. 
SCORM),  they  do  not  design  and  develop  learning  objects  at  present. 

Evaluation,  the  weakest  point  Both  Michael  and  Brad  reported  that  evaluation  is  the  weakest  part  in 
the  whole  process  of  customizing  training.  They  had  a sense  that  true  evaluations  should  take  place  on  the  job  a 
certain  amount  of  time  after  a training  program  is  completed. 

“I  think  a true  evaluation  would  be  six  months  to  year  down  the  road  on  how  the  person’s 
performance  on  the  jobs  changed  to  reflect  on  what  they  learned.  So  that  would  entail  some 
follow-up  survey  or  interviews.  1 don’t  know  they  do  that.  From  the  whole  point  of  training  I 
would  say  down  the  road.  Probably  that  happens  very  rarely  now  (Michael).” 

When  customization  takes  place?  There  are  two  phases  where  customization  takes  place.  Design 
and  development  phase  is  one  and  implementation  phase  is  another.  The  participants  reported  primarily  about 
design  and  development  of  customized  training.  However,  Stella  and  Eugene  mentioned  that  customization  can 
occur  when  individual  learners  plan  individual  training  plans  based  on  diagnosis  or  assignment  from 
supervisors. 

In  the  latter  case,  there  is  customization  of  training  plans  based  on  learner’s  profiles  including  required 
job  competencies,  training  history,  educational  back  ground,  and  so  on.  In  Eugene’s  company,  there  is  a 
learning  management  system  (LMS)  that  has  a competency  diagnosis  tool  for  assessing  the  gap  between  the 
current  competency  and  the  expected  competency,  as  well  as  for  prescribing  learning  solutions  to  fill  out  the 
gap  based  on  the  diagnosis.  At  this  phase,  training  programs  in  training  repositories  can  be  combined 
dynamically  to  customize  their  learning  plan  relevant  to  their  needs. 

Conclusion 

Through  in-depth  interviews  and  follow-up  interviews,  several  noticeable  facts  were  identified.  First, 
corporate  training  designers  customize  training  because  1)  training  should  impact  the  bottom  line,  2)  target 
audience  of  training  is  very  diverse,  3)  each  organization  has  own  ways  that  cannot  be  addressed  by  external 
training  solutions  and  4)  training  will  not  work  unless  it  is  customized. 

Based  on  collected  data  from  the  interviews  and  from  document  analysis , we  identified  four  different 
themes  to  describe  aspects  of  training  that  training  designers  customize.  They  are  content,  learning/instructional 
strategies,  timing  for  training,  and  location. 

In  order  to  maximize  customization  of  training,  training  developers  consider  business  needs,  and 
workgroup’s  needs,  organizational  learning  culture,  collective  learner  characteristics,  individual  learner  profile, 
content  characteristics  and  context  in  which  learners  apply  what  they  learn. 

Most  of  the  participants  shared  the  sense  that  they  are  not  fully  satisfied  with  their  current  customized 
training  and  that  there  is  much  room  for  more  successful  customization.  Even  though  they  customize  training  by 
considering  principal  playing  factors  identified  from  needs  analysis,  learner  analysis,  and  environment  analysis, 
they  all  reported  they  are  still  lacking  comprehensive  and  rigorous  analysis  owing  to  time  and  cost  constraints 

As  the  major  elements  of  design  process  for  customized  training,  they  mentioned  typical  ADDIE  steps. 
However,  they  emphasized  a comprehensive  and  thorough  target  audience  and  stakeholder-involved  process 
and  front-end  analysis. 

Training  designers  seem  to  have  common  issues  in  developing  customized  training.  They  stressed  that 
it  is  really  hard  but  critical  to  get  help  from  stakeholders  and  there  is  always  design  trade-off  by  means  of 
seeking  equilibrium  between  investment  and  return.  Also,  they  stated  that  they  should  avoid  assuming  what 
variables  are  at  play  in  training  since  assumptions  are  not  the  result  of  analysis. 

Limitations 

This  study  adopted  case  study  research  methods  proposed  by  Yin  (2003)  for  inquiring  unexplored 
questions.  Since  the  number  of  participants  was  small,  the  conclusions  of  this  study  may  not  be  generalizable  to 
all  corporate  contexts.  In-depth  interviews  with  more  participants  in  more  companies  in  diverse  industries 
would  make  this  study  more  rigorous  by  guaranteeing  external  validity.  Also,  the  average  length  of  experience 
as  training  designers  is  around  3 years.  It  would  be  desired  to  have  subjects  with  more  experiences. 
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The  interview  protocol  was  reviewed  by  one  peer  and  two  experts  in  order  to  have  face  validity.  The 
sources  of  data  were  multiple  (interview,  follow-up  interview,  and  document  analysis).  However,  it  would  be 
better  if  data  analysis  were  done  by  two  more  people  beyond  the  author  for  ensuring  reliability  and  validity.  For 
future  study,  triangulation  should  be  sought  in  terms  of  the  source  of  data  and  data  analysis. 


1 ASTD  (American  Society  for  Training  and  Development)  defined  training  as  follows:  “Training  focuses  on 
identifying,  assuring,  and  helping  develop,  through  planned  learning,  the  key  competencies  that  enable 
individuals  to  perform  their  current  job.  Training’s  primary  emmphasis  is  on  individuals  in  their  work  roles 
(McLagan,  1989,  p. 3).” 

2 

They  defined  customized  learning  as  “presenting  just  the  right  material  to  the  learner  on  demand.” 

3 

A conceptual  framework  is  a set  of  interrelated  concepts,  principles  and  ideas  that  help  organize  and  direct 
thinking  about  a concept  understudy.  It  is  used  to  assist  understanding  of  the  concept,  which  is  usually 
represented  with  graphics.  The  conceptual  framework  which  can  include  all  elements  related  to  customized 
training  and  can  depict  relationships  among  those  elements,  is  useful  in  designing  customized  training  and 
development.  The  conceptual  framework  helps  training  designers  to  understand  elements  to  be  considered  in 
designing  and  to  see  those  elements  from  a systemic  and  systematic  view. 

4 

Kevin’s  company  has  quite  comprehensive  rubric  for  analyzing  this  aspect:  1)  Size:  Is  the  target  audience  is  large  or 
small?  2)  Location:  Is  it  decentralized  or  centralized?  3)  Accessibility  to  Training:  Is  it  poor  or  good?  4)  Skill  Level:  Is  it 
diverse  or  uniform?  5)  Motivation:  Is  it  high  or  low? 
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What  to  Customize  ■ What  to  Consider 


What  to  Learn 
: Content 

How  to  Learn 
: Instructional 
Strategies 

When  to  Learn 
: Timing 

Where  to  Learn 
: Place 

•Scope 

•Level  of  difficulty 
•Details 
•Sequence 
•Scenarios 

•Learning  Activities 

(simulation,  Scenario- 
based,  Goad-based, 
Action  learning) 

•Locus  of  control 

(instructor -led, 

Facilitated,  Self-paced) 

•Modality 

•Performance  Support 
Supplement  (Job-aid, 
EPSS,  Mentors) 

•Mode  of  Delivery  (Web 
CD-ROM,  Video/  Print) 

•Flexible  training 
Schedule  forconsi 
dering  maximum 
Effectiveness 
•Just-In-Time 
training 

•On-the-job 

•Off-the-job 

(Corporate  Univ. 
External  training 
center) 

Individual  Learner  Profile 

(Role/  Competency/  Skill  & Knowledge/  Motivation/  Learning  Styles/ 
Career  Path/  Location/  Accessibility/  Ed. Background) 


Collective  Learner  Characteristics 


Context  in  which  learners  apply  what  they  learn 


Content  Characteristics 


Organizational  Learning  Culture 


Business  Objectives,  Department  Needs 


Figurel.  A conceptual  framework  of  customized  training 
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Abstract 

The  purpose  of  this  case  study  was  to  analyze  the  characteristics  of  an  online  learning  community  from 
the  perspectives  of  18  adult  learners  all  of  whom  completed  an  online  master’s  degree  program  in  instructional 
design  and  technology.  This  program  was  taught  at  a distance  using  the  Blackboard.com  e-learning  system. 
Severed  program  characteristics  supported  meaningful  learning  including  institutional  cooperation,  students’ 
prior  experiences  with  technology,  positive  peer  and  instructor  interactions,  constructivist  approaches  to 
teaching  and  learning,  cognitive  changes  generated  through  text -based,  asynchronous  critical  discourse, 
accessibility  and  reliability  of  web-based  technologies,  and  perspective  transformations  fostered  by  authentic 
assessments. 


Introduction 

This  paper  disseminates  the  preliminary  results  of  an  innovative  case  study  conducted  by  an 
interdisciplinary  team  of  six  faculty  members  who  teach  in  an  online  graduate  program  within  a college  of 
education.  They  investigated  the  question,  “How  was  an  effective  online  learning  community  (OLC)  developed 
among  the  first  cohort  of  students  in  the  Master’s  of  Science  in  Instructional  Design  and  Technology  program?” 
A WASC  accredited  program  inaugurated  in  2001  by  a large  state  university  in  Southern  California,  the  MSIDT 
program  focuses  on  the  direct  applications  of  technology  for  teaching,  learning,  and  curriculum  development  for 
professionals  in  K-12,  business,  industry,  military,  and  corporate  settings.  The  program  was  designed  to 
provide  students  with  a solid  background  in  the  field  of  instructional  design  with  an  emphasis  on  the  design  and 
creation  of  computer-based  training  and  Internet  technologies.  The  program  involves  faculty  from  elementary, 
secondary,  special  education,  reading,  and  educational  leadership  departments  all  of  whom  have  expertise 
and/or  training  in  instructional  technology,  curriculum  design,  adult  learning,  assessment  and  evaluation. 
Students  complete  the  30 -unit  program,  consisting  of  ten  online  courses  completed  over  a 20  months,  by  taking 
two  courses  per  16-week  term  segment  to  fulfill  their  degree  requirements.  A face-to-face,  two-day  orientation 
session  termed  “Boot-up  Camp”  and  a one-day  “Mid-point  Symposium”  provided  opportunities  for  community 
building.  This  presentation  is  geared  to  higher  education  faculty  and  administrators,  dis  tance  educators,  e- 
learning  facilitators,  web  developers,  instructional  designers,  students  of  instructional  design  and  technology, 
and  corporate  trainers  who  may  find  the  information  applicable  for  improving  their  professional  practices. 

Purpose  and  Rationale 

Online  learning  communities  exhibit  various  features,  characteristics,  and  purposes.  It  is  difficult  to 
categorize  the  attributes  of  online  learning  communities  using  standardized  educational  frameworks.  The 
purpose  of  this  study  was  to  examine  how  an  online  learning  community  emerged  from  the  first  cohort  of 
MSIDT  students.  While  several  contemporary  studies  have  explored  how  community  evolved  within  the 
context  of  a university  course  taken  online  for  a semester,  very  little  research  currently  exists  regarding  how  an 
online  learning  community  evolved  within  the  curricular  scope  and  sequence  of  an  online  degree-granting 
program.  Furthermore,  while  several  studies  have  investigated  how  an  online  learning  community  evolved 
from  one  researcher’s  perspective,  few  studies  have  integrated  the  multiple  perspectives  of  six  faculty  members 
who  teach  in  the  program. 
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Methodology 


Participants. 

The  participants  were  eighteen  adult  learners,  8 female  and  10  male,  all  of  whom  were  college 
graduates  working  in  educational  fields  including  K-12,  postsecondary,  corporate  training.  Most  students  lived 
in  the  state  but  a few  resided  outside;  no  students  from  foreign  countries  participated.  Longitudinal  research  on 
the  program  is  ongoing  and  the  preliminary  findings  presented  in  this  paper  will  be  expanded  to  incorporate 
data  collected  and  analyzed  from  two  additional  cohorts  of  approximately  18-25  students  each. 

Faculty. 

Each  faculty  member,  including  the  program  coordinator,  identified  a research  focus  area— a “unit  of 
analysis.”  All  focus  areas  reflected  theoretical  propositions  about  the  characteristics  of  online  learning 
communities  in  education  found  in  the  current  professional  literature.  The  following  focus  areas  regarding 
online  community  development  were  researched:  students’  attitudes  and  perceptions  about  learning  online; 
institutional  support,  accessibility  and  reliability  of  web-based  technologies;  the  impact  of  gender,  race, 
ethnicity,  disability  and  social  class  on  critical  discourse  and  the  social  construction  of  knowledge; 
constructivist  approaches  to  teaching  and  learning,  assessment/evaluation  of  learning  and  perspective 
transformation. 

Setting. 

The  context  of  the  research  was  a large,  urban,  comprehensive  BA  and  Master’s  Degree  institution  in  a 
western  state.  The  university  had  approximately  33,000  students  and  1,800  faculty.  Structurally,  the  MSIDT 
online  community  mirrored  the  university  community.  This  online  extension  of  the  university  was 
multidimensional  and  multilayered  given  the  broad  range  of  institutional  support  provided.  The  program  was 
inclusive  of  adult  learners,  faculty,  administrators,  support  staff,  curriculum  and  instruction,  and  technology 
resources  governed  by  university  policies  and  practices.  The  MSIDT  program  was  conceived  of  as  a “pilot 
project”  initiated  by  the  president  and  vice-president  of  the  university  and  represented  the  university’s  first 
distance  education  program  offering. 

Design  and  Analysis. 

The  case  study  method  proved  to  be  an  appropriate  research  design  because  it  incorporated  a 
comprehensive  research  strategy  that  linked  all  data  collected  to  the  initial  question  of  the  study.  According  to 
Yin  (2003),  a case  study  is  an  empirical  inquiry  that: 

investigates  a contemporary  phenomenon  within  its  real- life  context,  especially  when  the 
boundaries  between  the  phenomenon  and  context  are  not  clearly  evident,  and  that  relies  on 
multiple  sources  of  evidence,  with  data  needing  to  converge  in  a triangulating  fashion,  and 
as  another  result  benefits  from  the  prior  development  of  theoretical  propositions  to  guide 
data  collection  and  analysis  (p.13-14.). 

The  following  data  sources  were  converged  to  analyze  the  data  collected: 

1.  Secondary  Source  Data.  Co-researchers  consulted  secondary  source  materials  generated  from  the 
design,  delivery,  and  evaluation  phases  integral  to  program  development.  Co-researchers  reviewed 
documents  previously  prepared  for  national,  regional,  university,  and  college  committees  including 
advisory  councils,  workgroups,  faculty  presentations,  the  program  website,  Academic  Senate  approval, 
and  WASC  accreditation.  At  the  programmatic  level,  co-researchers  reviewed  course  proposals  and 
syllabi  and  the  online  content  of  the  ten  courses  comprising  the  program:  Hardware  and  Authoring 
Environments  (IDT505);  Research  Practices  in  Instructional  Design  and  Technology  (EDEL511); 
Instructional  Design  Issues  for  Technology-based  Instruction  (IDT520);  Instructional  Approaches  to 
Learning  and  Cognition  (IDT525);  Planning,  Designing,  and  Evaluating  Technology-based  Instruction 
(IDT530);  Instructional  Strategies  Pre-K  through  Adulthood  (IDT535);  Web-based  Teaching  and 
Learning  (IDT540);  Emerging  Technology  and  Issues  in  Instruction  (IDT545);  Practicum  in 
Instructional  Design  and  Technology  (IDT550);  and  Master’s  Project  (IDT597).  Co -researchers  had 
access  to  most  courses  which  contained  approximately  15  asynchronous  threaded  discussion  forums 
facilitated  by  the  course  instructor. 

2.  Discussion  Board  Archives . Data  was  collected  an  analyzed  from  the  discussion  board  transcripts 
from  four,  fifteen  week  courses  (IDT525,  IDT535,  IDT540,  IDT545).  This  database  served  as  the 
primary  data  source  for  the  study.  The  data  set  included  approximately  60  discussion  forums  with 
average  postings  of  125  postings  per  forum  totaling  7,500  student  postings.  Each  discussion  forum 
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contained  all  of  the  messages  that  shared  a common  overall  topic,  while  multiple  discussion  threads 
within  those  topics  reflected  specific  conversations  consisting  of  multiple  messages  that  addressed 
specific  subtopics.  The  distinct  advantage  of  using  archived  transcripts  was  to  reduce  participants’ 
reactions  to  the  presence  of  investigators. 

3.  Students’  Written  Records.  During  the  mid -point  symposium  students  responded  in  writing  to  the 
prompt,  “Being  an  Online  Student.’’  Responses  were  collected  and  made  available  to  the  research 
group  as  a data  source. 

4.  Focus  Group  Data.  A focus  group  for  students  was  conducted  by  an  outside  evaluator  in  two,  one- 
hour  sessions  split  into  two  parts:  a brief,  ten-minute  questionnaire  followed  by  a forty-five  minute 
discussion  about  students’  perceptions  about  the  strengths  and  weaknesses  of  the  program.  Both 
sessions  were  tape  recorded  with  the  consent  of  the  participants.  A faculty  focus  group  was  als  o 
conducted  in  a single,  seventy -five  minute  session  and  was  tape  recorded  with  participants’  consent. 
Results  of  both  evaluations  were  provided  to  the  students  and  to  the  faculty/co-researchers. 

5.  Student  Survey.  A web-based  interview  protocol  was  developed  and  posted  within  a week  of  students’ 
completion  of  the  MSIDT  degree.  The  survey  was  designed  to  elicit  thoughtful,  reflective,  and  in- 
depth  explorations  of  students’  perceptions  about  the  impact  of  the  program,  especially  their 
perceptions  about  learning  within  the  context  of  an  online  learning  community.  The  survey  questions 
were  grounded  in  the  theoretical  propositions  regarding  online  learning  communities.  It  consisted  of 
fourteen,  open-ended  questions  and  one  question  consisting  of  21  items  configured  in  a Likert-like 
scale  ranging  from  “strongly  agree,  agree,  neither  agree  or  disagree,  strongly  disagree,  and  no 
response.’’  Fifteen  out  of  the  eighteen  students  submitted  the  confidential  survey  for  a response  rate  of 
83%. 


Data  Analysis. 

Each  researcher  sorted  through  and  chunked  the  data  from  the  sources  indicated  above  from  the 
vantage  point  of  their  unit  of  analysis.  Each  researcher  identified  phenomenological  themes  relating  to  the  unit 
of  analysis  they  researched,  substantiated  with  verbatim  quotes  that  provided  thick,  rich  descriptions  of 
students’  perceptions.  Provisional  findings  from  each  thematic  analysis  were  reported  to  the  larger  group.  The 
pooled  information  was  then  reflected  on  by  the  whole  group  to  identify  patterns  regarding  the  phenomenon  of 
learning  within  an  online  community  as  it  was  experienced  by  MSIDT  students.  Trustworthiness  was  arrived  at 
through  data  triangulation,  empirical  and  consensual  validation  through  member  checks,  group  dialogue  and 
discourse,  and  through  critiques  of  multiple  drafts  of  research  reports  disseminated  to  members  by  the  lead 
author.  (A  multimedia  presentation  at  the  AECT  2004  conference  and  submission  of  this  paper  to  the  AECT 
Proceedings  provides  additional  opportunities  for  public  testing  of  the  preliminary  results). 

Preliminary  Findings 

Several  researchers  share  a common  interest  in  communities  enabled  by  the  Internet  and  seek  to 
redefine  community  in  a virtual  world.  Online  learning  communities  (OLC),  e-Learning  communities,  virtual 
communities,  and  communities  of  practice  are  terms  most  often  encountered  in  the  literature.  Online  learning 
communities  in  higher  education  are  communities  existing  in  virtual  environments  consisting  of  formally  and 
systematically  organized  teaching  and  learning  activities  in  various  academic  domains  in  which  the  instructor 
and  learners  are  geographically  separated  and  use  computer- mediated  technology  to  communicate.  According 
to  Rovai  (2001),  “strong  feelings  of  community  increase  the  flow  of  information  among  all  learners,  the 
availability  of  support,  commitment  to  group  goals,  cooperation  among  members,  and  satisfaction  with  group 
efforts”  (Bruffee,1993;  Dede,1996).  The  following  themes  discussed  below  emerged  as  constituitive  of  how 
learning  was  experienced  within  the  MSIDT  online  learning  community.  When  asked  to  review  definitions  of 
OLC’s  and  respond  to  the  survey  question,  “I  feel  I have  been  a member  of  an  online  learning  community  in  the 
MSIDT  program,”  93%  of  the  students  responded  that  they  strongly  agreed  or  agreed  with  this  statement  with 
one  student  affirming  in  writing  that  “a  true  learning  community  formed.”  Comments  culled  from  students’ 
open-ended  responses  characterized  the  MSIDT  online  learning  community  as:  “...a  group  of  individuals 
sharing  resources  and  learning  together  for  a common  purpose;”  “. . .coming  together  in  the  context  of  a 
primary  program  structure  and  Blackboard  where  communication  happens  through  a variety  of  means;”  “...an 
amorphous  being  that  grows  and  expands  depending  on  a variety  of  factors  including  the  personalities  of  the 
people  involved;”  “. . .participation  from  students  who  engage  in  a process  of  learning  that  requires  traveling 
over  and  revisiting  the  same  path  many  times;”  “. . .a  group  of  learners  who  gain  knowledge  in  an  online 
environment  that  allows  for  more  personal,  meaningful  interaction;”  However,  while  students  almost 
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unanimously  agreed  that  a community  had  formed,  there  was  a caveat.  One  student  observed:  “I  found  it  less 
meaningful  than  a place-based  commu  nity.” 


Prior  Experience  with  Technology 

Before  being  admitted  into  the  program,  each  student  submitted  a resume,  autobiography,  and 
completed  an  interview.  Experience  with  computers  was  required  for  admission  including  MS  Office, 
navigating  the  Internet,  using  e-mail  to  send,  receive,  attach,  and  download  messages.  However,  some  students 
in  the  first  cohort  had  additional  or  advanced  skills  in  programs  like  Macromedia  Director  and  Flash.  These 
students  were  identified  early  on  by  their  peers  and  served  as  informal  mentors  in  courses  where  authoring  skills 
were  a requirement.  Students’  characterized  the  community  as  being  comprised  of:  “educational  specialists 
and  technology  specialists. . .all  contributing  to  a holistic  understanding  of  the  ID  field;”  “team  leaders, 
technical  experts,  learning  experts,  production  volunteers;”  “members  relied  upon  for  certain  technical  skill;” 
“the  online  technical  person,  the  devil’s  advocate,  the  lost  soul,  and  many  people  that  were  often  supportive, 
helpful,  or  creative.”  Data  seems  to  reveal  that  students  in  the  cohort  recognized  and  valued  each  member’s 
contribution  to  the  community,  whether  or  not  they  were  advanced  technology  users.  Students’  comfort  with 
their  own  technology  skills  (prior  computer  experience)  may  have  made  it  easier  not  to  feel  intimidated  by 
another’s  advanced  skills.  This  supports  the  notion  that  students’  prior  experience  with  computers  can  boost 
positive  perceptions  of  online  learning,  as  noted  by  previous  researchers  (Huang,  2000).  These  prior  computer 
experiences  may  contribute  to  students’  perceptions  of  a developing  community  of  learners. 

Peer  Interaction 

Researchers  in  previous  studies  found  that  students’  satisfaction  with  online  learning  environments  is 
strongly  related  to  the  amount  of  active  interaction  with  other  learners,  noting  that  small  group  activities  can 
enhance  learning  motivation  (Jung  et  al,  2002;  Shin,  2003).  Creating  a safe  learning  environment  through 
positive  social  relationships  can  support  these  interactions  and  contribute  to  community  development.  Data 
collected  during  this  study  support  this.  Students  identified  several  strengths  including  social  posting  threads, 
group  projects,  group  discussions,  and  face-to-face  meetings  at  the  orientation  and  at  mid-point.  Students’ 
comments  included: 

• “Each  member  of  the  cohort  was  an  integral  part  of  the  learning  community.  We  all  came  to  the 
program  with  our  own  strengths  and  weaknesses  and  looked  to  the  community  to  fill  in  the  missing 
areas.  Each  member  challenged  me  to  become  my  best  by  asking  questions  and  commenting  on  my 
discussion  responses.” 

• “The  boot-up,  midpoint  and  commencement  experiences  were  necessary  since  the  relationships  needed 
to  be  established  in  order  to  gain  a sense  of  trust  with  peers  and  instructors.  I don’t  feel  I would  have 
felt  as  much  of  a sense  of  drive  completing  the  program  had  I not  established  relationships  with  my 
peers  and  instructors  early  on  by  meeting  face-to-face.  It  would  have  been  much  easier  to  quit  halfway 
through  if  it  had  all  been  online.” 

• “Social  relationships,  especially  online,  are  important  for  learning  to  occur,  in  that  the  exchange  of 
experiences,  ideas,  and  prior  knowledge  is  more  natural  when  all  members  feel  socially  connected  to 
the  community.” 

Most  students  (74%)  used  social  networking  to  decrease  their  sense  of  isolation.  Overwhelmingly,  students 
reported  a sense  of  inclusiveness  and  support  from  their  peers  via  their  interactions  through  discussion  boards, 
working  in  groups,  peer  evaluation,  and  e-mail.  Comments  included: 

• “Interaction  on  the  discussion  board  is  what  made  inclusion  happen.” 

• “Most  students  made  me  feel  included.  We  asked  each  other  for  ideas  and  we  were  always  flattered  to 
share  our  knowledge.” 

• “Private  e-mails  that  were  either  jokes  or  social,  unrelated  to  a specific  assignment  contributed  to  the 
feeling  of  inclusion.” 

• “I  made  sure  to  work  with  different  members  throughout  the  program  so  I did  not  feel  excluded.” 

While  positive  interactions  and  relationships  contributed  to  students’  satisfaction  toward  online  learning,  only 
26%  stated  that  the  most  meaningful  learning  came  about  through  their  interactions  with  others.  This  supports 
research  by  Jung  et  al  (2002)  that  found  collaborative  interaction  may  increase  students’  sense  of  community, 
but  it  does  not  necessarily  increase  learning  achievement.  Conversely,  negative  interactions  lead  students  to 
feel  excluded  and  may  decrease  their  desire  to  continue  in  the  program.  For  example,  one  student  noted  feeling 
excluded  by  one  classmate  who  snubbed  a request  for  help.  The  student  noted:  “Needless  to  say,  I did  not  e- 
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mail  [that  person]  ever  again.”  Another  student  expressed  that  complaining,  name  calling,  and  finger  pointing 
closed  down  learning  and  inhibited  community  participation. 

Teacher/Student  Interaction 

Students’  positive  interactions  with  their  instructors  influenced  their  perceptions  of  online  learning  and 
contributed  to  the  development  of  a learning  community.  Researchers  have  contended  that  the  instructor’s 
presence  and  social  interaction  influence  students’  motivation,  course  engagement,  and  learning  achievement 
(Jung  et  al,  2002;  Shin,  2003).  Data  collected  from  the  MSIDT  cohort  reflect  this.  Several  students  noted  that 
the  lack  of  instruction  on  an  authoring  tool  during  their  first  term  made  them  feel  excluded,  overwhelmed,  and 
intimidated.  On  the  other  hand,  students  noted  that  when  professors  “were  extremely  positive  and  encouraging” 
their  motivation  increased.  Instructional  factors  that  contributed  to  community  development  included 
responding  to  students’  contributions  in  weekly  summaries  of  discussion  topics,  constant  presence  on  the 
discussion  boards,  supportive  phone  calls,  and  a “good  response  time  for  e-mails  and  the  thoughtfulness  and 
caring  they  exhibited.” 

Constructivist  Approaches  to  Teaching  and  Learning 

Effective  online  learning  communities  are  founded  on  social  constructivist  pedagogy.  As  such,  the 
interchange  among  students  is  vital  to  a constructivist  learning  environment  where  the  conditions  conducive  to 
the  development  of  community  are  created.  In  this  program,  systematically  organized  teaching  and  learning 
experiences  engaged  learners  in  knowledge  construction  through  multiple  interactions  in  online  discussion 
groups.  Student -to-student  interactions  were  certainly  as  important  as  student-to-instructor  interactions  within 
this  community,  and  perhaps  more  so.  A learning  curriculum  was  co-created  by  the  learners  and  teachers  in  this 
program  and  became  a pool  of  resources  from  which  everyone  could  draw.  This  can  be  contrasted  with  a 
teaching  curriculum,  which  often  limits  distance  students  by  structuring  the  resources  and  controlling 
participants’  access  to  them.  The  difference  between  a learning  curriculum  and  a teaching  curriculum  can  be 
likened  to  the  difference  between  a successful  online  community  and  a correspondence  course.  In  a true  online 
learning  community,  knowledge  is  co-created  by  members  of  the  learning  community,  with  each  person 
contributing  his  or  her  additional  resources  to  the  “curriculum”  of  the  course.  This  can  be  contrasted  by  a 
correspondence  course,  where  students  simply  access  the  existing  course  curriculum,  respond  to  it,  and  submit 
assignments  individually.  Certainly  current  research  (Johnson,  2001;  Rogers,  2000)  suggests  that  students  from 
online  learning  communities  may  come  to  learn  more  from  the  information  added  by  the  class  members  than 
what  was  originally  presented  by  the  instructor.  Survey  data  reveals  that  86%  of  the  students  confirmed  they 
were  engaged  in  constructivist  learning  experiences.  A student  in  one  course  reflected:  “The  sum  was  much 
greater  than  the  total  of  the  parts  in  this  class.”  Another  commented:  “The  resources  we  saw  this  week  were 
awesome!  I learned  so  much  from  the  links  provided  this  week.  Thanks  everyone  for  your  contributions.  I can 
definitely  say  that  I benefited  from  all  of  your  professional  experiences  in  this  topic — something  I knew  little 
about  before  we  began  this  module.” 

Meaningful  learning  in  this  program  seemed  to  come  about  as  the  result  of  students’  interactions  with 
each  other,  rather  than  through  students’  individual  learning  efforts.  Comments  included: 

• “I  thought  I’d  feel  alone,  but  instead,  I feel  a part  of  something  different.  I guess  I feel  connected  to 
the  people  in  this  learning  community.  The  people  are  what  keep  me  going.” 

• “When  I first  started  this  program  I didn’t  consider  or  know  anything  about  an  online  community.  It 
became  apparent  eventually,  of  course.  And  actually  without  it,  I am  not  sure  I could  complete  this 
program,  as  there  have  been  times  when  I was  feeling  burned  out  and  overwhelmed  and  wondered  if  I 
could  sustain  the  effort  with  other  things  going  on  my  life.” 

• “I  also  feel  that  as  a learner  I have  been  intimately  involved  in  the  learning  process,  and  I have  a lot  of 
learner  control  as  well  as  input.” 

The  representative  quotes  highlighted  above  suggest  that  indeed,  as  Lave  and  Wenger  (1991)  suggest, 
learning  is  a relationship  among  people.  In  fact,  according  to  the  students  in  this  program,  the  “social  process, 
includes,  indeed  it  subsumes,  the  learning  of  knowledgeable  skills”  (p.  29).  The  “social  transactions”  among  all 
of  the  participants  in  this  OLC  allowed  all  members  to  see  themselves  as  legitimate  members  of  a community 
of  practice — or  an  online  learning  community.  This  is  a crucial  consideration,  especially  in  light  of  Conrad’s 
(2002)  recent  work,  which  suggests  that  the  creation  of  an  online  learning  community  serves  as  the  foundation 
for  a successful  learning  environment.  Other  research  (Brown,  2001)  certainly  emphasizes  the  important  point 
that  students  can  overcome  feelings  of  being  alone  when  they  support  one  another  in  an  OLC.  Moreover,  the 
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feeling  of  connection  to  the  learning  community  is  especially  important  because  students  who  feel  connected  to 
learning  communities  often  place  a higher  priority  on  the  class  and  spend  more  time  devoted  to  course  content 
(Brown,  2001). 


Accessibility  and  Reliability  of  Technology 

One  student  observed,  “I  believe  that  even  with  the  logistical  issues  we’ve  had  to  deal  with,  I have 
learned  a lot  in  this  program.  One  thing  that  I have  definitely  learned  is  that  the  creation  of  a community  of 
learning  is  very  powerful.  By  reading  others’  posts  I have  had  the  opportunity  to  ask  myself  some  very  deep 
questions.”  53%  of  the  students  surveyed  responded  that  they  agreed  or  strongly  agreed  that  when  there  were 
technical  difficulties  with  the  reliability  of  the  technology  their  sense  of  community  was  diminished;  43%  said 
if  they  could  not  access  the  course  their  sense  of  community  was  impaired.  However,  73%  of  the  students 
agreed  that  Blackboard  was  easy  to  navigate  and  this  reinforced  their  sense  of  community.  In  online  learning 
communities  technology  accessibility  and  reliability  is  a critical  factor  for  learning  and  community 
development.  If  access  to  the  technology  is  interrupted  or  if  the  technology  is  unreliable  or  slow,  students 
experience  frustration  which  inhibits  community  participation. 

In  this  study,  there  were  a few  technology  issues  having  to  do  more  with  interrupted  access  to 
Blackboard  and  occasional  slow  transmission,  but  the  technology  itself  was  not  a major  factor  impeding 
students’  learning.  Effective  and  frequent  communication  on  the  part  of  the  instructor  and  to  other  students  was 
more  important  than  the  technology.  A mix  of  instructional  strategies  matched  to  the  content  and  a variety  of 
online  tools,  especially  discussion  forums , were  equally  important  to  students’  success.  Students  with  low-tech 
skills  enjoyed  learning  the  software  programs  such  as  FrontPage  and  Dreamweaver;  high-tech  students 
apparently  enjoyed  an  open-ended  approach  to  learning.  Learning  in  a web-based  environment  was  especially 
beneficial  to  MSIDT  students  because  they  were  immersed  in  the  very  technologies  they  were  studying.  “I 
think  our  Masters  program  is  more  valid  since  we  are  becoming  experts  in  instructional  media  by  using  it  to  get 
our  degree,”  a student  remarked.  Instructors  anticipated  negative  comments  about  the  technology,  but  found 
that  students  were  generally  supportive,  or  at  least  neutral,  about  learning  in  a computer- mediated  environment. 
The  quality  of  the  instruction  and  frequency  of  communication  seemed  to  minimize  technology  issues.  Overall, 
students  affirmed  that  learning  via  technology  contributed  to  their  sense  of  community  as  these  quotes  reveal: 

• “Technology  can  help  or  hurt  education.  It  is  just  a tool  as  we  have  seen  many  time  s during  our 
MSIDT  program.  If  students  are  sitting  at  cubicles  working  on  computers  for  hours  on  end  this 
certainly  increases  isolation.  Students  who  are  creating  a multimedia -learning  object  together  as  a 
group  project  would  experience  a decrease  in  isolation.  Similarly  technology  could  increase  or 
decrease  the  level  of  abstraction  in  today’s  learning  environment.” 

• “Communication  in  the  online  classroom  is  important  in  order  to  maintain  a sense  of  community  and 
reduce  the  feelings  of  isolation  that  students  might  feel.  The  online  courses  that  I have  taken  took 
advantage  of  different  modes  of  communication  such  as  e-mail,  discussion  boards,  group  forums,  and 
real  time  chatrooms.  Providing  different  forums  in  which  students  and  instructors  could  interact 
ensured  timely  feedback,  privacy,  and  the  opportunity  to  discuss  issues  related  to  the  course.” 

• “I  started  this  program  believing  that  the  various  media  (audio,  video,  hyperlinks,  etc.,)  were  the 
powerful  tools  to  cause  learning.  Now  I see  that  these  media  are  simply  the  vehicles  for  delivering 
information  that  is  packaged  using  principles  of  instructional  design  that  match  the  learners.” 

Critical  Discourse 

The  survey  revealed  that  80%  of  students  agreed  that  engaging  in  the  online  discussions  challenged 
them  to  think  critically.  Students  perceived  that  open  dialogue  was  an  equalizing  force  and  alleviated  power, 
gender,  race,  class,  disability  and  cultural  diversity  issues.  “Online  discussions  are  a gift  from  heaven  for  me..  .1 
feel  free  from  cultural  mores  and  more  confident  expressing  my  ideas.  I feel  safe  to  convey  my  thoughts  in 
writing  because  I have  time  to  re-write  and  edit  my  posts  before  I submit  them,”  a student  remarked. 

Participants  shared  their  struggles  with  learning  new  information  because  they  had  established  a comfortable 
rapport  with  their  peers. 


Institutional  Cooperation 

Five  individuals  were  interviewed  to  ascertain  their  perceptions  about  community  development  as  it 
pertained  to  application  processing,  bookstore  ordering,  being  a program  liason,  coordinating  scheduling, 
creating  a curriculum  database  in  SIS+,  using  library  information  technology,  consulting  on  web  application 
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design  and  development,  and  serving  as  program  librarian.  54%  of  the  support  providers  surveyed  strongly 
agreed  or  agreed  that  the  MSIDT  online  learning  community  seemed  to  be  an  extension  of  the  campus 
community  online;  20%  did  not  agree  or  disagree;  and  20%  disagreed.  All  institutional  support  providers 
strongly  agreed  that  they  contributed  to  the  development  of  the  online  learning  community.  Likewise,  all 
strongly  agreed  or  agreed  that  they  were  comfortable  sharing  information,  knowledge,  suggestions  and  ideas 
with  the  program  coordinator,  MSIDT  students,  and  faculty.  They  expressed  commitment  to  the  program  by 
assisting  students  to  attain  their  academic  goals.  60%  of  the  support  staff  strongly  agreed  that  they  were 
participating  in  a constructivist  learning  environment  where  they  could  be  called  upon  to  help  students  solve  ill- 
structured  problems;  40%  believed  they  worked  collaboratively  with  the  students,  faculty,  and  program 
coordinator  to  achieve  the  program’s  outcomes.  Fully  80%  of  the  support  providers  strongly  agreed  that  they 
saw  themselves  as  an  important  and  respected  part  of  the  online  learning  community  while  20%  neither  agreed 
or  disagreed  with  this  statement. 


Perspective  Transformation 

Current  applications  of  technology  in  institutions  of  higher  education  are  not  taking  advantage  of  the 
potential  of  distance  learning  to  inspire  the  construction  of  new  models  and  outcomes  for  adult  learning. 
Moreover,  the  capacity  of  web-based  instruction  to  provide  models  of  transformative  learning  has  yet  to  be 
explored.  The  online  classroom  is  “fertile  territory  for  transformative  learning”  (Palloff  & Pratt,  p.  131). 
Mezirow  (1991)  defines  perspective  transformation  as: 

...the  process  of  becoming  critically  aware  of  how  and  why  our  assumptions  have  come  to 
constrain  the  way  we  perceive,  understand,  and  feel  about  our  world;  changing  these 
structures  of  habitual  expectation  to  make  possible  a more  inclusive,  discriminating,  and 
integrative  perspective;  and,  finally,  making  choices  or  otherwise  acting  upon  these  new 
understandings  (p.167). 

Perspective  transformation  occurs  along  four  lines  of  action:  a change  in  the  individual’s  existing 
frames  of  reference;  the  individual’s  ability  to  assimilate  entirely  new  perspectives,  a change  in  the  individual’s 
ability  to  be  more  critically  reflective  through  problem  solving  activities,  and  the  individual’s  ability  to 
construct  new  meaning  perspectives  or  habits  of  mind.  In  this  study,  87%  of  the  students  experienced 
perspective  transformations,  as  measured  through  reflective  self-assessments,  critical  discourse  in  the  discussion 
boards,  tests,  midterms,  individual  and  group  papers  and  projects.  There  is  evidence  that  students  were  able  to 
elaborate  on  their  previously  existing  frames  of  reference  as  a consequence  of  community  participation  given 
these  comments: 

• “I  took  subjects  I had  learned  in  my  credential  program  to  a much  deeper  level  of  learning  and 
understanding  which  enabled  me  to  revisit  some  of  the  areas  I hadn’t  given  much  thought  to.” 

• “I’m  slowly  getting  beyond  the  basics  and  really  getting  into  more  of  a long-term  understanding  of  the 
concepts.” 

• “This  concept  of  learner-centered  communities  definitely  opened  my  eyes  because  it  supplied  the 
words  to  the  ideas  that  have  been  banging  around  in  my  head  for  a few  years  now.” 

• “The  goals  I had  at  the  beginning  of  the  program  have  remained  the  same,  but  I am  more  confident 
now.  Colleagues  are  already  looking  to  me  for  advice  and  I in  no  way  would  ever  have  expected  that.” 

Some  students  assimilated  entirely  new  frames  of  reference: 

• “The  constructivist  perspective . . .was  a new  learning  experience  for  me  and  a positive  one  at  that.  I 
had  to  admit  that  I kept  thinking,  ‘What  is  it  that  the  teacher  expects  of  me.’  Once  I let  go  of  that  view 
and  took  the  view  of  ‘What  do  I expect  of  me,’  then  I felt  less  anxiety.” 

• “I  recall  one  Saturday  afternoon  I sat  in  Starbucks  and  read  through  a number  of  chapters.  I was  so 
fascinated  that  every  now  and  then  I would  catch  myself  talking  out  loud,  saying  something  to  the  tune 
of  ‘curious,  I did  not  know  that!’  I sort  of  shook  my  head  and  realized,  I am  learning.  How  could  that 
be?” 

• “Before  this  program,  I had  never  built  a lesson  plan  and  I didn’t  know  where  to  start.  I built  a lesson 
plan  and  learned  a lot.  Ironically,  the  same  week  I built  my  lesson  plan,  I was  asked  to  do  one  at  work 
and  the  experience  in  the  program  gave  me  the  confidence  to  build  a lesson  plan  properly.  Without 
this  experience,  I honestly  do  not  feel  I would  have  been  successful  at  work.” 

Some  students  changed  their  perspectives  by  becoming  critically  reflective  of  their  assumptions  through 
problem  solving  activities: 
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• “I  was  convinced  that  writing  the  collaborative  research  paper  was  going  to  be  a bust  both  in  terms  of 
the  end  product  and  its  educational  value,  but  I was  wrong.  I really  benefited  from  the  interaction  with 
my  peers  on  a research  task  and  found  the  challenges  of  asynchronous  collaboration  very  instructive.” 

• “At  the  beginning  of  this  program,  I felt  like  an  Olympic  high  diver  preparing  to  climb  the  ladder  up  to 
the  diving  board,  mentally  preparing  myself  over  the  first  few  months,  gathering  the  knowledge  to  take 
the  leap  three  stories  down  to  a complete  immersion  in  a new  career.  At  the  midpoint  of  the  program,  I 
felt  the  rush  of  adrenaline  and  significantly  more  confidence  to  leap  off  the  ledge  as  I am  now  poised 
with  new  knowledge  which  will  help  me  make  a clean  break  into  the  water.” 

Some  students  transformed  their  habits  of  mind,  the  filters  they  used  to  interpret  the  meaning  of  their 
experience: 

• “I  am  becoming  more  and  more  aware  that  I am  not  aware  of  my  awareness  until  I need  to  be  aware  of 
it.” 

• “I’m  excited  about  the  transformation  that  is  beginning  to  happen  at  my  school  because  I can  see  an 
opportunity  to  influence  the  school  as  it  transitions  towards  the  adoption  of  a new  philosophy.  I 
thought  I would  receive  resistance  when  I discussed  it  with  the  group,  but  since  then  that  discussion  the 
group  has  operated  at  a higher  level  of  enthusiasm.” 

• “The  major  learning  that  has  taken  place  for  me  in  this  program  has  been  the  gradual  chipping  away  of 
my  ingrained  instructivist  nature.  After  many  years  of  teaching  using  a single  epistemology,  the 
constructivist  crack  is  beginning  to  open.  It  takes  me  a great  amount  of  time  to  shift  perspectives,  so  it 
will  be  interesting  to  see  if  my  students  need  time  to  make  a shift  also,  but  I believe  they  will.” 

• “The  major  thing  I forsee  beyond  commencement  is  a grand  unification  of  all  of  the  theories, 
principles,  and  knowledge  that  I have  gained  in  this  program.” 

Discussion 

At  this  point  in  time,  this  case  study  provides  provisional  answers  to  the  following  question:  “How 
was  an  effective  online  learning  community  (OLC)  developed  among  the  first  cohort  of  students  in  the  Master’s 
of  Science  in  Instructional  Design  and  Technology  program?”  Preliminary  results  indicate  that  various  program 
characteristics  were  conducive  to  the  development  of  an  online  learning  community:  institutional  collaboration, 
members’  prior  technological  proficiency,  positive  peer  and  instructor  interactions,  adopting  constructivist 
approaches  to  teaching  and  learning,  cognitive  growth  through  effectively  facilitated  critical  discourse, 
accessibility  and  reliability  of  web-based  technologies,  and  authentic  assessments  prompting  perspective 
transformations.  These  findings  are  significant  for  online  program  providers  in  higher  education  because  they 
underscore  the  importance  of  building  online  learning  communities  as  foundations  for  transformative  learning. 
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Abstract 

Many  researchers  have  paid  attention  to  the  potentiality  and  possibility  of  the  shared  mental  model 
because  it  enables  teammates  to  perform  their  job  better  by  sharing  team  knowledge,  skills,  attitudes,  dynamics 
and  environments.  Even  though  theoretical  and  experimental  evidences  provide  a close  relationship  between 
the  shared  mental  model  and  successful  team  performance,  there  has  not  been  much  consideration  of  the 
factors  of  the  shared  mental  model  that  can  show  a causal  relationship  between  it  and  team  performance.  Based 
on  the  reviews  of  existing  studies,  the  purpose  of  this  study  is  to  specify  what  factors  affect  the  shared  mental 
model.  For  further  research,  it  is  necessary  to  conduct  empirical  studies  that  validate  the  causal  relationship 
among  the  identified  factors  of  shared  mental  models  that  affect  team  performance. 

Introduction 

Salas,  Dickinson,  Converse,  and  Tannenbaum(1992)  define  the  term  ‘team’  as  “a  distinguished  set  of 
two  or  more  people:  1)  who  interact  dynamically,  interdependently,  and  adaptively  toward  a common  and 
valued  goal,  object,  or  mission;  2)  who  have  been  assigned  specific  roles  or  functions  to  perform;  and  3)  who 
have  a limited  life  span  of  membership.”  For  example,  a team  in  corporate  settings  consists  of  two  or  more 
persons  who  have  similar  knowledge,  skills,  and  backgrounds.  Team  mernb  ers  have  to  achieve  goals  with 
similar  missions  and  visions  of  the  team  to  achieve  better  performance.  Even  though  each  team  member  has  a 
specific  role  and  responsibility,  they  should  work  together  interdependently. 

To  become  a high  performance  team  it  has  been  argued  that  each  team  member  should  share  various 
factors  such  as  team  knowledge,  skills,  work  attitude,  as  well  as  team  dynamics  and  the  environment 
surrounding  them.  In  other  words,  team  members  should  have  a shared  mental  model  that  represent  information 
regarding  the  team  members’  knowledge,  skills,  attitudes,  and  behavioral  tendencies  to  perform  better  across  a 
lot  of  different  domains  (Griepentrog  & Fleming,  2003).  Terms  such  as  team  mental  models,  shared  mental 
models,  shared  frames,  teamwork  schemas,  transactional  memory,  and  socio  cognition  have  been 
interchangeably  used  by  researchers  to  explain  variance  in  team  development,  performance,  strategic  problem 
definition,  strategic  decision  making,  and  organizational  performance  (Klimoski  & Muhammed,  1994).  In 
short,  the  shared  mental  model  can  be  defined  as  a representation  of  shared  knowledge  regarding  the  team,  the 
team’s  objectives,  and  information  of  team  processes,  communication,  coordination,  adaptation,  roles,  behavior 
patterns,  and  interactions  (Klimoski  & Mohammed,  1994). 

A great  deal  of  research  has  been  done  on  shared  mental  models.  For  instance,  Cannon-Bowers  and  his 
research  colleagues  (1993),  Kraiger  and  Wenzel  (1997),  and  Klimoski  and  Mohammed  (1994)  have  all  provided 
extensive  reviews  and  research  on  shared  mental  models.  In  addition,  they  have  attempted  to  analyze  the 
relationship  between  shared  mental  models  and  team  performance.  Numerous  studies  have  shown  that  a shared 
mental  model  improves  team  coordination  and  performance  (Cannon-Bowers,  Salas,  and  Converse,  1993; 
Klimoski  & Mohamed,  1994).  In  addition,  the  shared  mental  model  has  potential  value  as  an  explanatory 
mechanism,  which  helps  team  members  to  understand  team  performance  by  explaining  how  effectively  team 
members  interact  with  one  another. 

Expert  research  has  also  shown  that  team  members  of  high  performance  teams  could  often  coordinate 
their  behaviors  without  the  need  to  communicate  (Cannon-Bowers  et  ah,  1993).  In  addition,  when  team 
members  share  knowledge,  it  enables  them  to  interpret  cues  in  similar  manners,  make  compatible  decisions,  and 
take  appropriate  actions  (Klimoski  & Mohammed,  1994).  A shared  mental  model  also  helps  team  members 
explain  other  members’  actions,  understand  what  is  going  on  with  the  task  develop  accurate  expectations  about 
future  member  actions  and  task  states,  and  communicate  meanings  efficiently.  Many  studies  have  been  done  to 
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gain  a better  understanding  of  shared  mental  models  and  the  causal  relationship  between  shared  mental  models 
and  team  performance  (e.g  Cannon-Bowers,  Salas,  & Converse,  1993;  Cannon-Bowers,  & Salas,  1998; 
Cannon-Bowers,  Janis  & Salas,  Eduardo,  2001). 

However,  there  has  been  little  research  about  what  factors  the  shared  mental  model  comprises.  It  is 
important  to  examine  the  factors  of  shared  mental  models  because  it  is  useful  to  investigate  specific 
relationships  between  the  shared  mental  model  and  team  performance  as  well  as  understand  the  general 
characteristics  of  shared  mental  models.  This  study  synthesizes  the  existing  studies  regarding  the  factors  of 
shared  mental  models  with  alternative  views. 


Factors  affecting  shared  mental  models 

Cannon-Bowers  and  Salas  (1997)  suggest  that  team  members  can  use  shared  mental  models  to  develop 
team  members’  “knowledge,  skills,  and  attitudes  required  for  effective  teamwork"  and  the  “understanding  of 
facts,  concepts,  relations,  and  underlying  foundation  of  information  needed  to  perform  tasks."  That  is,  a shared 
mental  model  consists  of:  1)  requisite  knowledge  that  is  required  for  the  team’s  effective  task  performance,  2) 
the  skills  and  behaviors  that  are  necessary  to  perform  the  task  effectively  and  3)  the  appropriate  attitudes  that 
promote  effective  team  performance  (Cannon-Bowers,  Tannenbaum,  Salas  & Volpe,  1995).  These  are  either 
specific  or  generic  depending  on  task  and  the  team. 

In  addition,  a team’s  dynamics  and  their  understanding  of  a complex  situation  at  a certain  time  are  also 
considered  as  factors  of  shared  mental  models  (Cooke,  Stout,  & Salas,  1997;  Stout,  Cannon-Bower,  & Salas, 
1996).  The  dynamics  of  a team  are  mainly  team  interactions  that  happen  among  team  members  to  create  new 
ideas  and  facilitate  communication.  The  environment  surrounding  the  team  affects  the  shared  mental  model  as 
well  as  team  performance.  The  outside  environment  changes  the  role,  tasks,  and  final  goal  of  the  team  members 
so  that  they  have  to  adapt  themselves  to  new  environments  by  transforming  the  shared  mental  model  that  they 
used  to  have. 

Based  on  literature  reviews  on  the  shared  mental  model,  five  components  were  identified  and  selected 
as  main  factors.  The  factors  of  shared  mental  models  and  specific  descriptions  are  as  follows. 


Factor  1:  Team  knowledge 

Team  knowledge  consists  of  two  different  types  of  knowledge:  ‘teammate  knowledge’  and  ‘task 
knowledge’.  First,  teammate  knowledge  is  the  extent  of  knowledge  to  which  team  members  know  other  team 
members’  preference,  strengths,  weaknesses  and  tendencies  in  order  to  maximize  performance  (Cannon-Bowers 
et  ah,  1993).  This  type  of  knowledge  should  help  team  members  to  compensate  for  one  another,  predict  each 
other’s  actions,  provide  information  before  being  asked,  and  allocate  appropriate  tasks  or  roles  according  to 
member  expertise.  That  is,  as  members  become  more  aware  of  one  another,  they  can  adjust  their  own  behavior 
in  accordance  with  what  they  expect  from  teammates.  Knowledge  of  teammates  is  probably  more  useful  across 
a variety  of  tasks  rather  than  a single  task. 

Second,  task  knowledge  is  closely  related  in  specific  knowledge  that  is  needed  for  conducting  the  task. 
In  the  case  of  procedural  tasks,  team  members  should  share  task  models  that  are  declarative  and  procedural 
knowledge  for  conducting  the  tasks.  Whenever  tasks  are  conducted  in  unpredictable  ways,  the  value  of 
teammate  and  task  knowledge  becomes  more  important.  Cannon-Bowers  et  al  (1993)  argued  that  these  two 
types  of  knowledge  lead  team  members  to  better  performance  in  specific  tasks  by  sharing  teammates’ 
knowledge  and  skills  as  well  as  task  knowledge. 


Factor  2:  Team  skills 

Skills  refer  to  the  abilities  to  do  things  associated  with  successful  job  performance,  whereas  knowledge 
indicates  facts  and  information  essential  to  performing  a job  or  task.  Numerous  studies  have  addressed  some 
types  of  skills  that  are  needed  to  function  effectively  as  a part  of  a team.  For  instance,  Dickinson  and  McIntyre 
(1997)  identified  seven  core  skills  of  teamwork:  1)  communication,  2)  team  orientation,  3)  team  leadership,  4) 
monitoring  team  performance,  5)  feedback,  6)  backup  behavior  and  7)  coordination.  These  skills  are  important 
components  for  performing  tasks  successfully.  Cannon-Bowers  also  presents  a framework  for  team 
competencies  in  terms  of  team  specific  skills  and  team  generic  skills  (Cannon-Bower  et  al.,  1995). 

Team  specific  skills  depend  on  the  particular  team  and  include  specific  members’  ways  of  dealing  with 
conflicts  and  team  cohesion.  Team  generic  skills  are  transportable  to  other  teams.  These  competencies  include 
communication  skills,  interpersonal  skills,  and  leadership  skills.  Each  skill  is  not  independent,  but  involves 
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interactive  relationships. 

Factor  3:  Team  Attitudes 

Attitude  is  defined  as  “an  internal  state  that  influences  an  individual’s  choices  or  decision  to  act  in  a 
certain  way  under  particular  circumstances”  (Cannon-Bowers  et  al.,  1995).  This  attitude  in  shared  mental  model 
research  covers  shared  belief,  shared  value,  teamwork,  team  concept,  collective  orientation,  collective  efficacy, 
and  shared  vision.  First,  shared  belief  refers  to  the  category  of  what  we  believe  in.  Team  members’  shared 
beliefs  help  teammates  to  have  compatible  perceptions  about  the  task  and  environment  and  ultimately  reach 
effective  decisions.  Sub  factors  include  shared  beliefs  (Cannon  and  Edmondson,  2001)  and  cognitive  consensus 
(Mohammad  et  al.,  2000). 

Second,  shared  values  are  what  we  are  willing  to  work  for  (Carr,  1992).  High  performance  teams  have 
both  a clear  understanding  of  the  goal  to  be  achieved  and  a belief  that  the  goal  embodies  a worthwhile  or 
important  result.  Then,  when  everyone  is  clear  on  what  they  value,  they  can  set  goals  consistent  with  these 
values  and  the  team  will  believe  that  the  goals  are  worthwhile  and  important. 


Factor  4:  Team  dynamics 

Smith-Jentsch  and  his  colleagues  (2001)  examined  team  dynamics,  which  refer  to  the  understanding  of 
the  components  of  teamwork  that  are  critical  for  effective  performance  as  well  as  the  relationships  between 
these  components.  In  this  study,  team  dynamics  are  viewed  as  the  combination  of  dynamic  processes  of  team 
coordination  and  team  cohesion.  First,  team  coordination  consists  of  implicit  and  explicit  coordination  (Entin,  & 
Serfaty,  1999;  Stout,  Cannon-Bowers,  Salas,  & Milanovich,  1999).  In  explicit  coordination,  team  members 
communicate  to  articulate  their  plans,  actions  and  responsibilities,  whereas  implicit  coordination  describes  the 
ability  of  team  members  to  act  together  without  the  need  for  overt  communication.  For  successful  implicit 
coordination,  team  members  should  have  a shared  understanding  of  the  situation  and  an  accurate  understanding 
of  each  other’s  tasks  and  responsibilities . 

The  advantages  and  disadvantages  of  implicit  and  explicit  coordination  depend  on  the  nature  of  the 
task  and  the  task  environment  (Fiore  Salas,  & Cannon-Bowers,  2001).  The  ability  to  coordinate  implicitly  can 
provide  an  advantage  to  team  members  during  periods  of  intense  task  load  by  reducing  the  communication 
overhead  needed  for  coordinated  action.  Implicit  coordination  is  associated  with  effective  performance  if  team 
members  have  an  accurate  understanding  of  each  other’s  needs,  responsibilities,  and  expected  actions;  and 
communication  may  be  necessary  to  build  that  understanding. 

Second,  team  dynamics  include  team  cohesion,  defined  as  “a  dynamic  process  which  is  reflected  in  the 
tendency  for  a team  to  stick  together  and  remain  united  in  the  pursuit  of  its  instrumental  objectives  and/or  for 
the  satisfaction  of  members’  affective  needs”  (Carron  & Hausenblas,  1998).  Team  cohesion  includes  both  the 
task  and  social  aspects  of  cohesion.  Widmeyer,  Brawley,  & Carron  (1985)  pointed  out  that  a conceptual  model 
of  team  cohesion  should  be  distinguished  between  the  individual/team  and  task/social  cohesion.  The  four 
categories  of  team  cohesion  are;  1)  team  integration  relating  to  the  bonding  of  the  team  as  a social  unit,  2)  team 
integration  relating  to  the  task,  3)  individual  attractions  to  the  team  relating  to  personal  involvement  and  social 
interaction  with  the  team  and,  4)  individual  attractions  to  the  team  relating  to  the  team  task. 


Factor  5:  Team  Environment 

For  the  most  part,  the  team  environment  - which  is  external  conditions  affecting  the  formation  of  the 
shared  mental  model  - includes  technology,  organization,  synchrony  and  geographic  dispersion.  First, 
technology  is  a mechanism  that  affects  how  teams  interact  (McGrath  & Hollingshead,  1994).  One  example  of 
technology  is  information  technology.  Information  technology  is  a means  for  people  to  communicate  with  each 
other  and  transfer  information  through  networked  computing  systems.  There  are  numerous  studies  showing  that 
information  technologies  affect  dependencies,  information  flow,  and  workflow  among  collaborators  in  team 
performance  (Grinter  et  al.,  1999;  Sproull  & Kiesler,  1991). 

Second,  organizational  factors  such  as  culture,  structure,  and  standard  procedures  also  affect  team 
performance.  Various  studies  have  addressed  that  organizations  are  social  systems  that  affect  (and  are  affected 
by)  how  technologies  are  used  (DeSanctis  & Poole,  1994;  Orlikowski,  1992),  all  of  which  can  affect  the  types 
of  dependencies  present  in  a task. 

Third,  synchrony  and  geographic  dispersion  also  affect  successful  team  performance.  Team 
performance  is  affected  by  time  flow  in  which  team  performance  occurs  synchronously  or  asynchronously . In 
addition,  the  place  where  team  performance  happens  is  different  by  being  either  co-located  (i.e.,  same  place)  or 
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geographically  dispersed  (i.e.,  different  places).  Thus,  four-folded  modes  of  team  performance  occur  when 
synchrony  and  team  location  are  considered  (Bullen  & Bennett,  1993).  Consequently,  synchrony  and 
geographic  dispersion  need  to  be  considered  when  studying  shared  mental  models  because  they  can  generate 
different  work  arrangements  with  different  resulting  sets  and  types  of  dependencies. 


Conclusion:  Toward  general  factors  of  shared  mental  models 

Shared  mental  models  include  knowledge  relevant  to  team  work  such  as  knowledge  of  team  member 
roles,  responsibilities,  knowledge  of  teammates'  knowledge,  skills,  abilities,  beliefs,  preferences,  and  style  as 
well  as  knowledge  relevant  to  task  work,  such  as  cue-strategy  associations,  understanding  of  task  procedures, 
and  knowledge  of  typical  task  strategies  (Cannon-Bowers,  et  ah,  1995).  From  the  review  of  shared  mental 
models,  we  identified  five  main  factors  affecting  successful  team  performance,  which  are  team  knowledge, 
skills,  attitudes,  team  dynamics,  and  team  environments. 


Figure  1.  Theoretical  Framework  of  Shared  mental  model 


Discussion  and  Implication 

The  existing  studies  support  the  general  hypothesis  that  shared  mental  models  can  be  useful  as  a 
measure  of  team  performance.  Although  shared  mental  models  are  strongly  predictive  of  team  performance, 
they  have  not  been  clearly  defined  in  terms  of  the  sub  factors.  The  purpose  of  this  study  was  to  identify  the 
factors  which  affect  shared  mental  models.  This  study  is  hypothetical,  it  is  therefore  necessary  to  conduct  an 
empirical  study  that  validates  the  causal  relationship  among  identified  factors  and  team  shared  mental  models. 
Also,  we  need  to  figure  out  how  these  factors  impact  team  performance  and  how  we  can  best  support  team 
performance  based  on  our  findings. 
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Abstract 

The  American  Society  for  Training  and  Development  (ASTD)  defines  human  performance  technology 
(HPT)  as  “ a systematic  approach  to  analyzing,  improving,  and  managing  performance  in  the  workplace 
through  the  use  of  appropriate  and  varied  interventions.  ” The  first  step  in  this  approach  is  the  performance 
analysis  ( Gilbert,  1 978).  In  this  step,  the  performance  technologist/consultant  works  collaboratively  with  the 
client  to  examine  the  current  situation.  Performance  gaps  or  deficiencies  are  identified  and  are  prioritized 
according  to  the  needs  of  the  client  (ISPI,  2004).  This  case  study  describes  a project  in  a graduate  level  HPT 
class  at  a Midwestern  university.  A team  of  graduate  students  was  formed  to  address  computer  security  issues 
for  the  client  organization.  The  project  team  implemented  a performance  analysis  process  ( Define , Analyze,  and 
Select)  as  described  by  Schaffer  and  Douglas  (2004).  This  process  incorporated  tools  and  frameworks  such  as 
the  Performance  Relationship  Map  (Robinson  and  Robinson,  1995)  and  the  Performance  Pyramid  (Wedman 
and  Graham,  1998,  2004).  The  significance  of  the  project  is  that  it  provided  a real  world  context  in  which  the 
project  team  and  the  client  could  learn  about  HPT  processes.  The  experiences  were  enriched  by  the  request 
from  the  client  to  continue  the  HPT  process  after  the  analysis  project. 

Introduction 

“Think  performance,  not  training!”  (Robinson  & Robinson,  1995,  p.  6).  Human  Performance 
Technology  (HPT)  is  a field  of  professional  practice  which  is  project-based  and  focused  on  workplace 
effectiveness  (ISPI,  2004;  Stolovitch  & Keeps,  1999).  The  application  of  procedures  is  derived  from  scientific 
research  and  professional  experience  and  is  applied  to  the  solution  of  practical  problems.  The  American  Society 
for  Training  and  Development  (ASTD)  defines  performance  technology  as  “a  systematic  approach  to  analyzing, 
improving,  and  managing  performance  in  the  workplace  through  the  use  of  appropriate  and  varied 
interventions.” 

Many  names  have  been  given  to  this  field,  including  human  performance  technology  (Stolovitch  & 
Keeps,  1999),  human  performance  improvement  (HPI)  (Rothwell,  Hohne  and  King,  2000),  human  performance 
enhancement  (Rothwell,  1996)  performance  consulting  (Robinson  & Robinson,  1995),  performance  engineering 
(Dean  1994;  Gilbert,  1978),  performance  technology  (Harless,  1992),  and  so  forth.  In  addition  to  the  various 
names,  many  different  performance  improvement  process  models  exist. 

The  first  step  in  the  HPT  approach  is  the  performance  analysis,  in  which  the  performance  technologist 
works  collaboratively  with  the  client  to  examine  the  current  situation  at  one  or  more  of  the  following  levels: 
societal,  organizational,  process,  work  group,  or  individual.  Performance  gaps  or  deficiencies  are  identified  and 
are  prioritized  according  to  the  needs  of  the  client  (ISPI,  2004). 

Schaffer  and  Douglas  (2004)  developed  a framework  for  object-oriented  performance  analysis  for  the 
Automated  Object-Oriented  Performance  Analysis  Project  (AOOPA).  This  framework  recommends  that 
organizations  not  bypass  the  problem-solving  process  by  neglecting  the  definition  and  analysis  of  a 
problem/opportunity  or  by  skipping  directly  to  the  selection  of  a single  solution.  According  to  this  framework, 
the  most  basic  elements  in  performance  analysis  are  three  iterative  phases;  define,  analyze,  and  select. 

The  major  tasks  for  each  phase  are  as  follows: 

■ Define:  Define  the  opportunity  or  problem 

o Start  with  clear  statement  of  the  opportunity  or  problem 
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o Identify  gaps  between  what  is  and  what  should  be  at  the  organizational  level 
o Communicate  with  stakeholders  using  visual  models 
0 Provide  a rationale  for  decisions 

■ Analyze:  Analyze  gaps  between  what  should  be  and  what  is  at  performer  level 

o Develop  a performance  model  as  foundation  for  development  of  data  collection  instruments  to 
identify  current  performance  and  work  environment  barriers 
o Review  data  with  clients  and  collaborate  to  identify  gaps 
o Provide  a rationale  for  decisions 

■ Select:  Select  solutions  and  recommend  actions 

o Map  possible  causes  for  gaps  to  possible  solutions 
o Collaborate  with  clients  to  prioritize 
o Blend  solutions  most  appropriate  to  organizational  context 
o Report  recommendations 
o Provide  a rational  for  decisions 

This  paper  presents  the  experiences  of  a team  of  graduate  students  in  Educational  Technology  at  a 
university  in  the  Midwest  while  conducting  a performance  analysis  in  a HPT  course  project.  The  team  utilized 
the  consolidated  performance  analysis  with  three  major  phases  - define,  analyze,  and  select;  and  integrated 
other  organizational  frameworks  and  models,  including  Robinson  & Robinson’s  (1995)  Performance 
Relationship  Map  and  Wedman  and  Graham’s  (2004)  Performance  Pyramid,  into  the  process  to  complete  the 
performance  analysis  of  computer  users  who  lived  in  residence  halls.  The  process  and  results  of  the 
performance  analysis,  as  well  as  challenges,  limitations  and  lessons  that  the  performance  analysis  team  learned 
are  discussed  in  this  paper.  The  team’s  completion  of  the  project  provided  them  with  the  experience  necessary 
to  answer  the  following  questions,  which  they  discovered  are  crucial  to  the  performance  analysis  process: 

• What  can  be  done  to  help  the  client  understand  the  performance  analysis  process?  What  actions 
can  be  taken  to  ensure  that  the  client  is  involved  in  performance  analysis? 

• When  analyzing  the  causes  of  the  performance  gaps,  what  can  be  done  to  encourage  clients  to 
investigate  causes  other  than  the  lack  of  skills  and  knowledge  (or  solutions  other  than  training)? 

• What  is  the  next  step  after  the  performance  analysis? 

The  Case 

Overview 

The  study  was  a class  project  in  a graduate  level  HPT  class.  A project  team  of  four  graduate  students 
with  two  doctoral  and  two  master’s  students  was  formed.  All  members  of  the  team  were  novices  in  the  HPT 
field.  The  major  concepts,  including  the  definitions  to  the  field,  systems  theory,  HPT  frameworks  and  models, 
and  performance  analysis  were  introduced  before  the  beginning  of  the  project.  The  client  organization  was  one 
of  the  groups  in  the  security  and  policy  department,  known  as  Security  Outreach  and  Training,  under  the 
computer  services  at  a midwestern  university.  The  client  was  responsible  for  proactively  and  reactively 
combating  the  information  security  problems  of  the  university  population. 

The  performance  analysis  process  began  in  October  2003  with  an  introductory  meeting  with  the 
manager  of  the  client  organization.  The  project  team  completed  the  performance  analysis  in  December  2003. 

Define  Phase 

An  initial  client  meeting  with  the  client  team  was  called  for  project  alignment.  The  project  team 
immediately  began  working  through  the  Performance  Relationship  Map  (Figure  1)  with  the  client  using  an 
interview  guide  that  addressed  key  performance  analysis  factors.  The  business  problem  was  identified  as 
“Insecure  residence  hall  computer  network  computers  are  generating  an  unnecessary  workload  for  the  security 
and  policy  department.”  The  business  need  was  defined  as  “The  university  community  will  have  a secure 
computing  infrastructure.  The  target  performers  were  identified  as  the  computer  network  users  who  lived  in  the 
residence  halls.  The  Operational  Results  and  the  On-The-Job  Performance  Should’s  on  the  Performance 
Relationship  Map  were  documented  after  the  desired  organizational  goals  and  desired  individual  performance 
goals  were  discussed. 
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Figure  1.  The  Performance  Relationship  Map  adapted  from  Robinson  & Robinson  (1995) 
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In  addition,  a system  model  (a  visual  representation  of  the  organizational  process  with  inputs  and 
outputs)  was  sketched  together  with  the  client  in  order  to  identify  all  the  key  stakeholders  and  their  relationships 
surrounding  the  business  problem.  Resources  and  available  support  from  the  client  were  also  discussed  and 
arranged,  such  as  the  available  data  for  the  current  operational  results.  Finally,  the  data  collection  process  was 
brainstormed  with  the  client. 

After  the  meeting,  a Statement  of  Work,  which  summarized  the  business  problem  and  need,  key 
stakeholders,  resources  for  the  performance  analysis,  target  performers,  and  initial  performance  gaps  based  on 
available  data  and  discussions  in  the  initial  meeting,  was  prepared  and  emailed  to  the  client  for  confirmation. 

Analyze  Phase 

Data  Collection  The  analyze  phase  started  with  the  analysis  of  available  data  provided  by  the  client. 
The  client  had  previously  collected  data  about  the  number  of  machines  compromised  in  particular  outbreaks  and 
vulnerabilities,  the  percentage  of  email  messages  received  by  the  target  performers  that  were  classified  as  spam, 
and  the  number  of  machines  in  the  mail  server  infected  by  a recent  virus.  The  project  team  used  this  data  to 
begin  completing  the  Operational  Results  -Is  box  on  the  Performance  Relationship  Map. 

Next,  due  to  the  population  size  of  10,500  performers,  requirements  for  quantitative  data,  and  the  need 
for  confidentiality  of  respondents,  the  project  team  selected  a questionnaire  as  the  instrument  for  additional  data 
collection.  In  addition,  a performance  model  (Robinson  & Robinson,  1995),  which  detailed  the  competencies 
and  computer  security  best  practices  for  the  performers,  was  drafted.  As  mentioned  by  Schaffer  (2000),  Dean 
and  Ripley  (1997)  indicated  that  the  ability  to  integrate  and  synthesize  useful  frameworks,  processes  and  data 
that  link  the  major  systems  and  subsystems  within  and  outside  the  boundaries  of  an  organization  is  a critical 
skill  set  of  performance  improvement  specialist.  The  development  of  the  questionnaire  is  an  example  of  this,  as 
it  required  the  integration  of  different  process  models  and  organizational  frameworks.  It  was  designed  and 
revised  based  on  the  discussions  during  the  initial  client  meeting,  the  initial  performance  relationship  map, 
existing  documents  developed  by  the  client,  the  performance  model,  as  well  as  the  comments  and  suggestions 
from  the  client  and  the  project  advisor. 

In  an  effort  to  simplify  the  data  analysis  process,  the  project  team  developed  the  questionnaire  so  that  it 
provided  insight  into  both  the  existence  and  also  the  causes  of  existing  performance  gaps.  To  achieve  this,  the 
questions  were  crafted  so  that  they  assessed  the  performers’  use  of  the  best  practices  and  also  emphasized 
several  of  the  building  blocks,  such  as  knowledge  and  skills,  tools  and  equipment,  etc.  in  the  Performance 
Pyramid  (Figure  2).  The  questions  that  addressed  the  Performance  Pyramid  blocks  provided  the  basis  for 
determining  the  environmental  barriers  and  the  internal  causes  on  the  Performance  Relationship  Map. 
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Figure  2.  The  Performance  Pyramid  (Wedman  & Graham,  2004) 


After  development,  the  project  team  sent  the  questionnaire  to  the  client  for  review.  The  client’s 
feedback  was  incorporated.  The  questionnaire  was  then  finalized  and  developed  into  a web-based  questionnaire. 
Because  the  target  performers  were  computer  network  users  who  lived  in  the  residence  hall,  and  because  the 
majority  of  the  network  users  living  in  the  residence  halls  were  undergraduate  students,  the  subjects  were 
chosen  to  be  the  undergraduate  students  from  a 100-level  computer  technology  class,  a 200-level  education 
class,  and  a 100-level  engineering  class.  The  survey  was  conducted  for  one  week  and  173  complete  responses 
were  collected. 


Data  Reporting  The  data  collected  were  processed  to  identify  current  individual  performance.  A 
summary  of  data,  pie  charts  and  bar  charts  were  prepared  for  the  data  review  meeting  as  visual  representations 
that  were  used  to  compare  quantities,  amounts,  and  proportions.  A Pareto  Chart,  which  is  a specialized  type  of 
bar  chart  that  organizes  the  data  from  highest  to  lowest  (or  lowest  to  highest),  was  also  used.  This  chart  helps  to 
determine  the  causes  with  the  most  impact;  and  “is  a highly  useful  way  to  establish  priorities  on  problems  or 
causes  by  surfacing  and  displaying  those  which  are  most  problematic”  (Rothwell,  Hohne  and  King,  2000,  p. 

79).  Instead  of  diagramming  the  responses  for  causes  in  Pareto  Charts,  the  gaps  were  diagramed  in  order  to  help 
identify  the  unacceptable  gaps.  The  causes  were  then  arranged  according  to  the  building  blocks  of  the 
Performance  Pyramid  for  the  ease  of  distinguishing  the  type  of  cause,  such  as  knowledge  & skills,  rewards  & 
incentives,  etc.  (Figure  3). 
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Figure  3.  Responses  to  Question  3:  “Please  tell  us  why  you  don’t  update  antivirus  software  at  least  once  per 
week”  grouped  by  Performance  Pyramid  block 
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it’s  worth  it.  (2.3%) 


Tools  & Environment 

• Q3_l_3:  I 
don’t  have  the 
software. 
(6.4%) 

• Q3_l_6: 1 
don’t  know 
how  to  get 
help.  (3.5%) 

• I don’t  have  it. 
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Data  Review  Meeting  During  the  data  review  meeting,  the  preliminary  data  report  with  a summary  of 
data,  descriptive  statistics,  i.e.  frequencies,  drafts  for  the  performance  model,  Pareto  charts,  the  performance 
relationship  map,  and  results  related  to  causes  organized  according  to  the  Performance  Pyramid  was  presented 
to  the  clients.  The  data  from  the  questionnaire,  as  well  as  the  data  that  was  previously  collected  by  the  client, 
were  used  to  complete  the  On-the-Job  performance  IS  data  on  the  Performance  Relationship  Map  (Figure  4). 

Through  discussion,  the  client  and  the  project  team  reached  a group  consensus  on  unacceptable  gaps 
and  environmental  barriers.  The  project  team  and  the  client  brainstormed  some  potential  causes  for  the  gaps  in 
addition  to  those  supported  by  the  data.  Frequently,  the  project  team  had  to  redirect  the  client  so  that  causes 
other  than  a lack  of  knowledge  or  skills  were  considered.  The  project  team  did  this  by  referencing  the 
Performance  Pyramid  and  asking  the  client  about  potential  causes  from  specific  blocks  other  than  knowledge 
and  skills. 

In  addition,  the  project  team  also  had  to  redirect  the  client  to  focus  on  the  causes  instead  of  the 
solutions  during  the  meeting,  as  the  client  representatives  desired  to  jump  straight  to  the  solutions  once  they  saw 
the  data  that  displayed  the  performance  gaps. 

After  the  meeting,  the  project  team  prepared  a cause  prioritization  worksheet,  which  listed  the  potential 
causes  for  the  performance  gaps,  and  provided  to  spaces  for  the  two  representatives  of  the  client  organization  to 
rank  those  which  they  felt  were  important  to  address.  Originally,  this  worksheet  was  to  be  discussed  during  the 
data  review  meeting.  However,  time  did  not  allow  for  this  discussion.  Instead  the  worksheet  was  emailed  to 
the  client.  Besides  requesting  that  the  client  provide  a rationale  for  the  prioritization  of  the  causes,  the  team  also 
asked  the  client  to  list  the  potential  solutions  if  applicable.  Unfortunately,  the  representatives  of  the  client 
organization  did  not  provide  much  of  a rationale  or  possible  solutions.  The  time  constraint  that  forced  the 
project  team  to  conduct  this  part  of  the  performance  analysis  by  email  negatively  affected  the  response  of  the 
client. 
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Figure  4.  Completed  Performance  Relationship  Map 


Organizational 

Individual  (ResNet  Users  in  Residence  Halls) 

Desired  Results  (Should) 

♦ 5%  or  less  of  all  machines  in  ResNet 
compromised  in  any  outbreak  of 
vulnerability. 

♦ The  mailhub  virus  statistic  goes  down  to 
zero. 

4 20%  of  all  emails  are  spam. 

Desired  Results  (Should) 

♦ 100%  of  users  have  strong  passwords. 

♦ 100%  of  users  change  security  settings  to  the  highest 
level  that  works  for  the  website  that  they  want  to 
browse. 

♦ 100%  of  users  disable  or  set  cookies  to  be  discarded 
when  a website  is  closed. 

♦ 100%  of  users  choose  not  to  download  unknown  files 
or  programs  when  browsing  the  Internet. 

♦ 100%  usage  of  firewall. 

♦ 100%  of  users  update  anti-virus  software  at  least  one 
per  week. 

♦ 100%  of  users  apply  software  and/or  system  updates 
and/or  patches  regularly. 

4 100%  usage  of  anti-virus  software. 

♦ 100%  of  users  do  not  give  out  email  addresses  or 
personal  information  to  suspicious  websites. 

GAP 

GAP 

Organizational 

Individual  (ResNet  Users  in  Residence  Halls) 

Current  Results  (Is) 

♦ Approximately  10-20%  of  all  machines  (or 
1000-1500  machines  of  10,  500  registered 
hosts)  on  ResNet  compromised  in  the 
outbreak  of  RPC  DCOM  worms. 

♦ Approximately  1000  hosts  were  infected  at 
peak  infection. 

♦ Approximately  40%  of  all  emails  are  spam 
(i.e.  400,000  of  1 million  of  messages 
received  per  day  in  the  Purdue  mailhub). 

Current  Results  (Is) 

(Based  on  data  collected  from  the  survey) 

4 8.1%  of  users  change  password  regularly  (Q6_6). 

4 36.4%  of  users  change  security  settings  to  the  highest 

level  that  works  for  the  website  that  they  want  to 
browse  (Q7). 

4 Strong  password: 

^ 41%  of  users  disable  guest  login 

(Q6_10). 

^ 42.2%  of  users  use  different 

passwords  for  different  accounts 
(Q6_7). 

^ 45.7%  of  users  don’t  set  computer 

to  remember  passwords  (Q6_8). 

^ 52%  of  users  use  keys  next  to  each 

other  on  the  keyboard  (Q6_4). 

^ 57.8%  of  users  don’t  set  computer 

to  automatic  login  (Q6_9). 

^ 65.9%  users  do  not  use  personal 

information  like  SSN,  birthday, 
names  and  etc.  (Q6_3). 

^ 68.2%  of  users  use  8 or  more 

characters  (Q6_l). 

^ 76.9%  of  users  don’t  share  password 

with  anyone  (Q6_5.) 

^ 83.2%  of  users  use  letters,  numbers 

and  other  characters  (Q6_2). 

4 42.2%  of  users  disable  or  set  cookies  to  be  discarded 

when  a website  is  closed  (Q8). 

4 50.9%  of  usage  of  firewall  (Q4). 

4 70.5%  of  users  update  anti-virus  software  at  least  one 

per  week  (Q3). 

4 85%  of  users  apply  software  and/or  system  updates 

and/or  patches  regularly  (Q5). 

4 86.1%  of  users  choose  not  to  download  unknown 

files  or  programs  when  browsing  the  Internet  (Q9). 

4 93.1%  of  usage  of  anti-virus  software  (Q2). 

4 94.8%  of  users  do  not  give  out  email  addresses  or 

personal  information  to  suspicious  websites  (O10). 
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Select  Phase 

Select  Solution  After  receiving  the  Cause  Prioritization  Worksheet  from  the  client,  the  project  group 
discussed  solution  types  that  would  be  the  most  appropriate  and  developed  solutions  accordingly.  Solutions 
were  brainstormed  with  a job  aid  for  matching  causes  with  possible  solutions  based  on  cause  type.  For  example, 
when  dealing  with  lack  of  knowledge  and  skills,  training,  job  aids,  and  feedback  systems  were  listed  as  the 
appropriate  solution  categories.  A list  of  proposed  solutions  to  the  client  can  be  found  in  Figure  5. 

Figure  5.  List  of  proposed  solutions 


List  of  Proposed  Solutions 

Cause  type:  Expectations  and  Feedback,  Desire  and  Self-Concept 
Solution  type:  Feedback  Systems 

■ Implement  a software  module  to  ensure  the  use  of  a strong  password 

■ Send  a personalized  email  alert  during  major  virus  outbreak  to  remind  performers  to  update  their 
anti-virus  software 

■ Send  a pamphlet  emphasizing  free  security  products  available* 

■ Send  a personalized  email  reminder  to  change  passwords  once  or  twice  during  the  semester 

■ Email  a monthly  news  release  that  describes  how  computers  were  compromised  and  the  resulting 
problems  due  to  weak  or  unchanged  passwords  or  lack  of  or  improper  firewall  use 

Cause  type:  Knowledge  and  Skills 
Solution  Type:  Training 

■ Deliver  a tutorial  when  the  performers  sign  for  their  ResNet  accounts  to  teach  about  anti-virus  software 
use  and  firewall  use* 

■ Present  ITaP  security  resources  and  security  best  practices  in  dorm  orientations* 

Solution  Type:  Job  Aids 

■ Include  a checklist  with  the  characteristics  of  a strong  password  on  the  “Change  Password”  screen 

■ Include  a security  best  practices  checklist  in  the  Purdue  Mortar  Board  (student  daily  planner) 

■ Include  a “how-to”  job  aid  in  the  personalized  email  reminder  to  change  passwords  (mentioned 
above) 

Cause  type:  Rewards  and  Incentives 

Solution  Type:  Reward  and  Recognition  Systems 

■ Give  away  free  gifts  (e.g.  mouse  pads,  can-holders  “koozies”)  with  security  best  practices 
checklists  printed  on  them* 

■ Allow  students  who  proceed  through  the  tutorial  (mentioned  above)  to  enter  a drawing  for  free 
computer  hardware  or  software* 

*The  performance  analysis  team  suggested  to  the  client  that  these  solutions  be  included  in  the  blended  solution  of 
holding  “Security  Awareness  Weeks”  at  the  beginning  and/or  end  of  each  semester 


The  potential  solutions  were  then  rated  by  three  of  the  four  team  members  based  on  the  following  four 
solution  selection  criteria:  opportunity  (an  organization-level  support  and  commitment),  capability  (the 
collective  knowledge  and  skills  of  an  individual,  department  or  organization),  collaboration  (the  level  of  user 
involvement  in  adoption,  adaptation,  and  implementation  processes),  and  motivation  (the  perception  or  attitude 
potential  adopters  and  stakeholders  have  of  the  attributes  of  the  solution).  Categories  were  rated  using  a 5-point 
scale  (1  for  Disagree  and  5 for  Agree).  In  addition,  the  estimated  cost  for  each  potential  solution  was  also 
estimated  using  a 5-point  scale  (1  for  the  most  cost-effective).  The  averages  for  the  ratings  from  the  three 
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project  members  were  calculated.  The  rationales  for  ratings  were  compiled  to  indicate  the  strength  and 
weakness  of  each  solution. 

The  culmination  of  completing  a performance  analysis  is  the  recommendation  of  solutions.  A variety 
of  solution  recommendations  and  alternatives  were  provided  due  to  an  assortment  of  causes  for  each 
organizational  gap.  As  noted  in  Figure  5,  the  team  developed  blended  solutions.  Blended  solutions  are  often 
more  effective  because  they  can  be  re-purposed  to  develop  collateral  materials  that  will  assist  the  performer 
when  transferring  or  apply  solutions  in  the  workplace,  and  short-term  as  well  as  long-term  solutions  can  be 
developed  (Schaffer  & Douglas,  2002). 

Final  presentation 

During  the  final  presentation,  the  project  team  reviewed  the  whole  process  of  the  project  as  a re- 
alignment strategy.  The  gap  and  cause  analysis  report  was  presented  with  the  recommendations  that  related 
systematically  with  the  potential  solutions.  Next,  the  potential  solutions  were  presented  with  a description  of 
how  the  solutions  were  selected,  rated,  and  blended  in  order  to  meet  the  business  needs.  Furthermore,  a list  of 
actions,  including  communications,  training,  and  work  environment,  were  provided  to  the  client  as  the  answers 
of  “What  next?”  or  “What  does  it  all  mean?”  Those  actions  were  presented  as  the  crucial  steps  and  processes 
that  should  be  considered  by  the  client.  Open  discussion  followed  for  the  client  to  select  the  potential  solutions. 

An  additional  constraint  surfaced  during  this  meeting  as  it  became  evident  that  the  client  organization 
did  not  want  to  exercise  too  much  control  over  the  target  performers.  The  client  representatives  made  it  clear 
that  they  were  not  willing  to  take  actions  such  as  revoking  the  user’s  computer  access,  even  if  the  performer  was 
not  acting  according  to  information  security  best  practices. 

The  outputs  of  this  performance  analysis  included  the  identified  performance  needs,  environmental 
barriers,  causes  and  needs  for  improvement.  These  outputs  provided  the  client  representatives  with  a glimpse 
into  attitudes  and  actions  of  their  target  performers.  In  addition  the  causes  that  were  identified  could  later  be 
used  to  facilitate  the  evaluation  of  solution  effectiveness,  since  reaction,  learning  and  skill  transfer  evaluation, 
as  well  as  cost-benefit  evaluations  could  be  related  to  these  causes. 

Overall,  the  client  representatives  were  pleased  with  the  solutions  presented,  as  evidenced  by  their 
asking  of  the  project  team  for  assistance  in  designing,  developing,  and  implementing  some  of  the  proposed 
solutions. 


Discussion 


Challenges  and  Constraints 

The  project  had  several  challenges  and  constraints.  The  target  performer  population  was  too  broad  and 
had  diverse  characteristics.  The  population  size  was  about  10,500  with  a variation  of  backgrounds  in  terms  of 
descriptors  such  as  major  areas  of  study  and  class  level.  Additionally,  due  to  the  time  constraint,  the  project 
team  had  to  shorten  the  data  collection  to  one  week  with  a focused  group  of  subjects  that  may  not  have  been  a 
representative  sample  of  the  population.  Therefore,  the  project  team  considered  the  questionnaire  as  a pilot 
study  that  could  be  continued  and  improved  upon  in  the  future.  Nevertheless,  the  data  collected  allowed  the 
client  and  the  team  to  interpret  the  data  with  effective  tools,  and  to  identify  individuals’  current  performance  as 
a baseline.  At  the  same  time,  the  client  was  able  to  learn  more  about  the  data-driven  systematic  process. 

As  Robinson  and  Robinson  (1995)  mentioned,  Block  (1981)  has  identified  three  consulting  styles  that 
are  used  with  frequency:  “the  pair-of-hands,  expert,  and  collaborative  styles”  (p.  18-22).  Only  the  collaborative 
style  will  yield  results  for  a performance  consultant.  The  team  experienced  “the  pair-of-hands”  throughout  the 
process  as  they  continually  attempted  to  keep  the  client  informed  about,  and  involved  in,  the  performance 
analysis  process.  In  addition,  on  some  occasions,  some  of  the  processes  were  modified  to  meet  the  client’s 
needs.  For  example,  the  project  team  could  not  prioritize  the  major  causes  of  gaps  during  the  data  review 
meeting.  The  cause  prioritization  was  conducted  by  emailing  the  client  a form  to  fill  out.  Though  the  client 
completed  the  form,  the  rationale  provided  was  below  the  project  team’s  expectations.  Similarly,  due  to  the  time 
constraint,  the  project  team  had  to  alter  the  original  plan  for  the  select  phase  which  should  have  included  two 
meetings  with  the  client  in  order  to  brainstorm,  present  and  rate  the  potential  solutions  with  the  input  of  the 
client.  Instead,  the  project  team,  independent  of  the  client,  rated  the  solutions,  and  then  presented  them  to  the 
client  during  the  final  meeting.  The  feedback  of  the  client  was  later  retrieved  via  email  from  the  course 
instructor. 
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Lessons  Learned 

Collaboration  is  the  Key  to  Success  The  most  important  lesson  that  the  team  learned  while  conducting 
the  performance  analysis  is  that  getting  the  client  involved  in  all  phases  of  the  analysis  process  greatly  improves 
the  process.  This  can  be  done  by  keeping  the  client  informed  of  the  project  status  via  e-mail,  status  memos, 
minutes,  and  agendas.  Again,  as  mentioned  by  Robinson  and  Robinson  (1995),  Block  (1981)  indicated  that  it 
will  bring  1 + 1 = 3 (synergy)  when  the  client  and  the  consultant  work  collaboratively.  In  addition,  keeping  the 
client  representatives  involved  and  sharing  HPT  visual  models  and  frameworks  with  them  helped  to  increase 
their  understanding  of  the  performance  analysis  process.  This  performance  analysis  experience  was  positive  for 
all  involved  because  of  the  involvement  of  various  stakeholders  throughout  the  entire  process.  Some  of  the 
feedback  from  the  client  is  as  follows: 

“I  would  give  both  groups  all  5's  across  the  board  for  Organization,  Professionalism,  Quality,  Results, 
and  Outcomes.  The  only  thing  I wish  we  could  have  had  more  time  and  thought  put  into  was  the 
survey  process.  But  given  the  constraints,  I think  this  was  well  done  also.” 

“The  computer  security  team  presented  us  with  some  useful  suggestions  and  I am  planning  on 
implementing  (or  at  least  trying  to  implement)  some  of  their  suggestions.  I even  spoke  with  our 
Communications  AVP  about  the  give-aways  and  she  thought  that  was  possible.” 

In  October  2004,  the  project  team  contacted  the  client  for  comments  about  the  effectiveness  of  the 
proposed  solutions.  Some  comments  are  as  follows: 

“We  are  in  the  process  of  getting  the  online  class  up  and  running.  We  are  working  with  a vendor  to 
obtain  prizes  as  suggested  by  your  group  to  entice  students  to  participate  in  our  course.” 

“I  think  there  is  some  improvement.  We  have  had  an  increase  in  the  number  of  downloads  of  our  free 
Anti-virus  software  from  our  web  site.” 

Visual  Models  are  Effective  The  performance  analysis  approach  was  introduced  to  the  client  during 
the  process.  The  use  of  visual  models,  such  as  the  Performance  Relationship  Map,  Performance  Pyramid,  and 
system  model  proved  to  be  effective  in  helping  the  client  understand  the  processes.  For  example,  the  use  of  the 
Performance  Pyramid  aided  the  project  team  in  helping  the  client  to  look  to  solutions  beyond  training  to 
solutions  that  addressed  gaps  other  than  those  caused  by  a lack  of  knowledge  and  skills.  The  client 
representatives  enjoyed  the  use  of  visual  models,  as  evidenced  by  the  following  feedback  they  provided  when 
asked  if  the  project  provided  any  opportunity  for  them  to  learn  about  the  HPT  approach: 

“Yes.  I think  the  most  interesting  concepts  were  breaking  down  the  lack  of  performance  into  the 
different  categories,  such  as  knowledge  & skills  verses  desire,  expectations  and  the  other  categories 
defining  why  someone  is  performing  or  not.” 

“I  think  the  visual  models  were  effective  in  showing  the  different  reasons  for  performance  gaps.  I 
found  that  breaking  the  questions  into  categories  that  specifically  addressed  each  of  the  areas  of  the 
pyramid  was  very  helpful  in  understanding  reasons  for  the  performance  gaps.” 

Conclusions 

During  completion  of  the  project,  the  team  gained  specific  knowledge  about  the  performance  analysis 
process.  They  learned  that  it  is  difficult,  but  possible  to  create  solutions  for  an  organization  that  has  little 
control,  or  that  chooses  to  exercise  little  control  over  the  target  performer.  In  addition,  the  team  learned  that  it  is 
often  necessary  to  remind  the  client  of  the  data  to  make  sure  that  the  process  remains  data-driven.  Furthermore, 
the  team  became  more  aware  of  information  security  issues  (the  context  of  the  project).  Finally,  the  project 
team  learned  that  visual  models  could  be  effective  tools  for  educating  clients  about,  and  involving  them  in  HPT 
processes. 

In  conclusion,  the  significance  of  the  project  is  that  it  provided  a real  world  context  in  which  the 
project  team  and  the  client  could  learn  about  HPT  processes.  The  novices  on  the  project  team  learned  how  to 
apply  HPT  processes  and  work  effectively  with  a client,  and  the  client  representatives  learned  how  to  assess  and 
address  performance  gaps  with  varied  solutions.  In  addition,  the  experiences  were  enriched  and  the  consultant- 
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client  relationship  continued  by  the  request  from  the  client  to  continue  the  HPT  process  after  the  analysis 
project  through  the  design,  development,  implementation,  and  evaluation  of  the  solutions. 
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Abstract 

This  study  was  designed  primarily  to  investigate  the  impact  of  anonymous  peer  feedback  on  student 
meaningful  learning  in  higher  education.  Forty-seven  students  from  three  undergraduate  classes  from  a central 
US  university  participated  in  this  study.  Students  were  asked  to  build  a web-based  project.  In  the  experimental 
group , technology-mediated  peer  review  and  feedback  were  provided  for  students  to  use  in  improving  then- 
projects  prior  to  instructor  assessment.  The  control  group  received  no  peer  feedback.  Students’  projects  were 
independently  evaluated  and  analyzed.  Results  indicated  that  there  was  no  significant  difference  on  project 
quality  between  the  control  and  experimental  groups.  However,  post-assessment  survey  indicated  that  students 
had  generally  positive  perceptions  of  this  process. 

Introduction 

Promoting  student  autonomy  and  encouraging  student  meaningful  learning  has  become  an  important 
focus  in  higher  education  in  recent  years.  When  students  take  a more  active  role,  learning  becomes  more 
meaningful  and  their  achievement  is  improved.  Researchers  (Orsmond  & Merry,  1996;  Orsmond,  Merry,  & 
Reiling,  2002)  argued  the  need  for  academic  staff  to  switch  their  roles  from  teaching  to  facilitating  learning  in 
order  to  achieve  higher  student  engagement  and  responsibility  and  suggested  that  in  assessment  practices,  some 
“power”  should  be  “handed  over”  to  students. 

Students’  behavior  and  attitude  toward  learning  are  shaped  by  the  assessment  system  (Freeman,  1995). 
To  achieve  the  outcome  of  meaningful  learning,  appropriate  assessment  methods  should  be  applied. 
Unfortunately,  the  traditional  instructor-led  assessment  method  provides  only  limited  opportunities  for 
assessment  and  feedback.  Peer  assessment  is  believed  to  be  one  of  the  solutions,  as  it  not  only  provides 
additional  feedback  but  also  stimulates  student  interaction  and  involves  students  in  thinking  critically  about 
assessment  criteria.  Within  this  context,  the  assessment  process  can  be  viewed  as  “the  learning  exercise  in 
which  the  assessment  skills  are  practiced.”  (Sluijsmans,  Brand-Gruwel,  & van  Merrienboer,  2002). 

Peer  assessment,  according  to  Topping  and  his  colleagues  (Topping,  Smith,  Swanson,  & Elliot,  2000),  is  a 
process  in  which  peers  evaluate  the  achievement  or  performance  of  others  of  similar  status.  Cheng  & Warren 
(Cheng  & Warren,  1999)  further  defined  this  assessment  form  as  reflection  on  “what  learning  had  taken  place 
and  how.”  Peer  assessment,  as  an  alternative  to  traditional  solo  instructor  assessment,  has  been  applied  in  higher 
education  courses  such  as  writing,  computer  science,  arts  and  engineering,  etc  (Liu,  Lin,  & Yuan,  2002).  There 
are  a number  of  studies  illustrating  how  this  process  can  be  applied  in  both  summative  and  formative 
evaluations.  The  majority  of  the  literature  on  peer  assessment  in  higher  education  has  focused  on  the 
“assessment  of  individual  contribution  to  group  work”  or  the  correlation  between  peer  rating  and  instructor 
rating  (Hanrahan  & Isaacs,  2001,  Sluijsmans  et  al.,  2002).  There  are  also  some  studies  exploring  the  perceptions 
and  feeling  of  students  towards  this  process. 

Peer  assessment’s  benefits  on  higher  thinking  and  cooperative  learning  have  been  established.  Pope 
(2001)  suggested  peer  assessment  stimulates  student  motivation  and  encourages  deeper  learning.  Freeman 
(1995)  argued  that  studying  the  marking  criteria  and  evaluating  peers’  work  can  improve  students’  critical 
assessment  skills.  Topping  (1998),  after  reviewing  109  articles  focusing  on  peer  assessment,  confirmed  that 
peer  assessment  yields  cognitive  benefits  for  both  assessor  and  assessee  in  multiple  ways.  Those  “benefits  might 
accrue  before,  during  and  after”  the  process.  He  further  concluded  that  feedback  yielded  from  this  process  has  a 
positive  impact  on  students’  grades  and  subjective  perceptions. 

Most  current  peer  assessment  methods  are  conducted  through  paper-based  systems.  Two  concerns 
associated  with  this  system  that  hinder  the  widespread  acceptance  of  this  process  are  anonymity  and  the 
administrative  workload. 

Researchers  noted  their  concerns  towards  the  anonymity  issue  in  peer  assessment  (Davies,  2002).  One 
assumption  of  this  process’s  credibility  is  that  students  usually  provide  fair  and  unbiased  feedback  to  their 
peers.  However,  as  reported  by  a number  of  studies,  students  find  it  difficult  to  rate  their  peers.  They  don’t  want 
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to  be  too  harsh  on  their  peers;  they  are  uncomfortable  critiquing  others’  work  (Hanrahan  & Isaacs,  2001; 
Topping  et  al.,  2000).  Conducted  in  an  open  environment,  potential  biases  like  friendship,  gender  or  race  could 
cause  students  to  rate  good  performance  down  or  poor  performance  up.  Instructors  need  to  design  and  maintain 
a distribution  system  to  keep  both  reviewers’  and  reviewees’  information  confidential  and  anonymous,  and  at 
the  same  time,  traceable  for  instructors  to  maintain  the  fluency  of  the  process. 

Taking  more  control  of  their  learning  process  motivates  students.  Assessing  peers’  projects  deepens 
their  understanding  of  the  topic  being  reviewed.  Constructive  feedback  from  peers  helps  to  reevaluate  and 
improve  their  own  performance.  These  steps  all  contribute  to  more  professional  performance.  At  the  same  time, 
one  of  the  advantages  of  peer  assessment  is  the  reduced  assessment  time  for  instructors.  The  instructor  will 
spend  less  time  diagnosing  the  underlying  problems  of  student  response,  providing  feedback,  and  reassessing 
students’  revisions.  This,  of  course,  is  good  news  for  instructors  who  face  the  pressure  brought  by  continuous 
growth  in  student  enrollment  and  limited  instructional  time.  However,  another  problem  might  be  raised  at  the 
same  time:  the  management  of  feedback  documentation  (Davies,  2002).  Hanrahan  and  Isaac  (2001)  reported 
more  than  40  person  hours  for  documentation  work  in  classes  with  244  students.  The  load  increases  with  larger 
classes.  This  is  one  of  the  major  reasons  some  researchers  found  this  process  time  consuming. 

Technology-mediated  peer  assessment  has  been  proposed  as  a solution  to  provide  anonymity  and 
minimize  the  workload.  In  this  system,  data  can  be  automated  and  summarized,  and  students  and  instructors 
have  instant  access  to  data  once  they  are  generated.  The  whole  process  can  be  conducted  in  an  anonymous  way 
via  the  Internet.  Reviewers  and  reviewees  are  not  aware  of  each  other.  However,  the  integration  of  technology 
in  peer  assessment  in  higher  education  is  still  at  an  early  stage  of  development.  Limited  data  are  reported  even 
though  various  forms  of  computer-assisted  peer  assessment  methods  have  been  described  (Topping,  1998).  Our 
study  addresses  this  issue  by  investigating  an  application  of  a peer  assessment  process  that  is  delivered  via  an 
anonymous  Web-based  feedback  management  system.  Our  interest  is  in  the  impact  of  technology-based  peer 
feedback  on  student  meaningful  learning  and  students’  perceptions  of  this  method  in  higher  education. 

In  this  study,  peer  assessment  and  feedback  were  utilized  only  for  promoting  learning,  not  as  a 
substitute  for  instructor  grading.  Its  three  critical  aspects  include:  1)  defining  assessment  criteria,  in  which 
students  think  about  what  is  required;  2)  evaluating  the  performances  of  peers;  3)  providing  constructive 
feedback  for  further  project  improvement.  Compared  with  other  methods  in  this  area,  this  study  is  innovative 
because  it  utilizes  a database-driven  peer  feedback  website  to  ensure  anonymity,  simplify  data  management  and 
stimulate  student  interaction. 

Based  on  the  outcome  of  previous  studies,  our  hypothesis  is: 

1.  Web-based  peer  feedback  engages  students  in  critical  thinking  and  promotes  meaningful  learning,  thus 
improving  project  quality. 

2.  Students  gain  positive  perceptions  about  this  process.  They  feel  the  process  promotes  deeper  learning 
and  helps  them  improve  their  project  quality. 

Facilitating  Website 

The  emergence  of  information  technology  and  rapid  increase  of  online  capacity  have  provided  a new 
arena  for  education.  Like  other  instructional  platforms,  innovative  methods  integrating  technology  have  been 
proposed  and  tested  in  the  assessment  field.  In  the  early  1990’s,  a novice  collaborative  learning  network  was 
studied  at  the  University  of  Liverpool,  England  (Rada,  Acquah,  Baker,  & Ramsey,  1993;  Rushton,  Ramsey,  & 
Rada,  1993).  One  feature  of  this  multi-user  database-driven  system  was  designed  for  facilitating  peer 
assessment.  This  cost-effective  tool  constructed  an  environment  where  students  could  easily  read,  grade  and 
provide  suggestions  to  each  other’s  work.  Although  this  system  presented  incomparable  superiority  in 
stimulating  students’  interaction  and  reducing  administrative  load,  as  noted  by  the  authors  (Rushton  et  al., 

1993),  this  process  was  not  anonymous.  Assessees’  identities  could  be  easily  revealed. 

Tsai  and  his  colleagues  (Tsai,  Liu,  Lin,  & Yuan,  2001)  employed  a peer  review  network  to  foster 
students’  critical  thinking  skills.  Students  completed  their  projects  and  uploaded  them  to  the  network.  This 
network  enabled  students  to  review  each  other’s  performance  and  provide  constructive  feedback.  Then  students 
revised  their  own  work  according  to  the  comments  from  peers.  This  procedure  was  repeated  two  or  three  times. 
Preliminary  observation  suggested  that  this  system  had  positive  influence  on  students’  assignments.  Tsai  further 
asserted  that  peer  assessment  supported  by  a network  was  the  most  effective 

Based  on  the  previous  research,  our  study  was  designed  to  ensure  anonymity  and  facilitate  the  peer 
review  process.  A database-driven  website  was  built  that  enabled  students  to  register  and  log  in  with  the 
username  and  password  they  specified.  This  system  contained  separate  interfaces  for  instructors  and  students 
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(Figure  1).  In  the  student  interface,  each  student  was  randomly  assigned  two  WebQuest  projects  created  by  two 
peers.  Once  students  logged  in,  they  could  perform  two  roles  — reviewer  and  reviewee.  As  reviewers,  they 
reviewed  the  two  assigned  projects  and  provided  their  feedback  confidentially  according  to  the  marking  criteria 
for  each  project.  The  data  were  summarized  for  the  author  of  each  project;  as  reviewees,  they  had  access  to  the 
feedback  for  their  own  projects.  The  instructor  interface  was  designed  to  enable  instructors  to  keep  track  of  the 
peer  review  process.  For  each  student,  the  instructor  had  access  to  the  two  reviews  created  by  the  student  as 
well  as  the  feedback  this  student’s  project  received  from  two  peers. 

Figure  1 


Student  Interface 


Instructor  Interface 


This  system  has  the  following  major  merits: 

1.  Anonymity  was  assured.  This  system  ensured  anonymity  in  two  ways.  First,  students’  identities  were 
coded  as  numbers.  No  personal  information,  such  as  initials  of  their  names,  could  be  associated  with 
their  work.  Secondly,  students’  projects  were  WebQuest  web  sites.  Since  they  were  typed  and  running 
on  the  Internet,  no  handwriting  would  reveal  their  identities  or  characteristics,  such  as  gender.  The 
potential  risk  of  gender  bias  demonstrated  by  Falchikov  and  Magin’s  study  (1997)  was  eliminated. 

2.  Management  workload  was  reduced.  All  the  data  were  aggregated  and  transmitted  from  users’ 
computers  to  database.  Management  workload  was  minimal. 

3.  Students’  interaction  was  stimulated.  Submitted  data  were  instantly  summarized.  Students  and 
instructors  had  immediate  access,  which  encouraged  students’  engagement  and  promoted  their 
interaction. 


Methods 


Subjects 

This  study  was  conducted  with  forty-eight  students  from  three  undergraduate  classes  at  a central  US 
university.  Although  two  teachers  instructed  these  three  classes,  the  same  procedure  was  followed.  Students 
were  all  from  the  same  course  entitled  “Instructional  Technology”  at  the  College  of  Education  and  Human 
Sciences.  Students  were  randomly  assigned  into  an  experimental  group  (27)  and  a control  group  (21).  One 
student  in  the  experimental  group  dropped  the  study  for  personal  reasons.  Since  this  course  is  a required 
technology  application  course  for  pre -service  teachers  at  a college  level,  students  have  different  academic 
backgrounds  and  range  from  freshman  to  senior. 

Procedure 

In  this  study,  students  were  asked  to  build  a WebQuest  project  and  upload  it  to  the  Internet.  A 
WebQuest  is  “an  inquiry -oriented  activity  in  which  most  or  all  of  the  information  used  by  learners  is  drawn 
from  the  Web”  (Dodge  & March,  1995).  This  model,  developed  by  Bernie  Dodge  and  Tom  March  in  the  early 
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1995,  is  designed  to  involve  users  in  a learning  process  of  analysis,  synthesis  and  evaluation,  which  promotes 
their  critical  thinking  and  scaffolding  skills. 

In  the  control  group,  students  were  asked  to  individually  develop  WebQuest  projects  by  themselves 
after  studying  the  content  area  and  the  assessment  criteria.  In  the  experimental  group,  the  following  five  stages 
were  involved  (Figure  2): 

Figure  2 


Experiment  Groups  Control  Groups 


Stage  1 : Studying  the  content 
area  and  discussing  assessment 
criteria 

Stage  1:  Studying  the  content 
area  and  discussing  assessment 
criteria 

i 

1 

Stage  2:  Developing  WebQuest 
project 

Stage  2:  Developing  WebQuest 
project 

1 

Stage  3:  Judging  the  performance 
of  peers  and  providing  feedback 

1 

Stage  4:  Reviewing  feedback  from 
peers  and  improving  projects 

1 

Final  Stage:  Project  submission 
to  the  instructors 

Final  Stage:  Project  submission 
to  the  instructors 

Stage  1:  Studying  the  content  area  and  discussing  assessment  criteria 

After  thoroughly  studying  the  content  area,  students  were  presented  a rubric  and  were  asked  to  study  it. 
Students  were  informed  that  this  was  the  evaluation  criteria  that  would  be  used  by  the  instructor  in  assessing 
their  projects  and  for  their  use  in  reviewing  peers’  projects.  It  depicted  the  basic  elements  required  for  a quality 
WebQuest;  thus  it  was  important  and  beneficial  to  the  assignment.  The  assessment  rubric  was  studied  in  two 
levels  in  a student-centered  atmosphere.  First  students  formed  groups  and  discussed  the  rubric;  then  they  were 
encouraged  to  share  their  understanding  in  class. 

Stage  2:  Developing  WebQuest  project 

Students  were  requested  to  make  a WebQuest  project,  build  it  a web  site,  and  upload  it  to  the  Internet. 

Stage  3:  Judging  the  performances  of  peers  and  providing  feedback 

The  website  built  to  facilitate  the  peer  review  process  was  introduced  to  students.  Once  students  logged 
onto  the  peer  feedback  website,  they  had  access  to  two  peers’  WebQuest  projects,  which  were  randomly 
assigned  to  them.  Students  were  asked  to  rate  the  projects  and  provide  detailed  comments  according  the  rubric 
criteria. 

Stage  4:  Reviewing  feedback  from  peers  and  improving  their  own  projects 

Feedback  from  peers  was  automatically  summarized  and  made  available  to  the  creator  of  each  project. 
After  viewing  the  peer  rating  scores  and  comments,  students  had  the  opportunities  to  go  back  to  improve  their 
own  projects. 

Final  Stage:  Project  submission  to  instructors 

Students  submitted  their  projects  to  instructors  for  grading. 
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Survey 


After  students  in  the  experiment  groups  submitted  their  final  projects,  they  were  asked  to  complete  a 
survey.  Twenty-two  students  in  the  experimental  groups  responded  to  this  survey.  The  survey  replicated  from 
previous  study  (Lin,  Liu,  & Yuan,  2002)  consisted  of  11  5-point  Likert  Scale  items  dealing  with  their  general 
perceptions  about  the  process,  as  well  as  two  open-ended  questions  related  to  their  likes  and  dislikes:  “Please 
specify  what  you  like  most  in  this  peer  assessment  procedure.”  “How  would  you  change  this  peer  assessment 
procedure?  And  why?” 

Scoring  Procedure 

Two  independent  raters  were  trained  and  each  of  them  graded  all  the  projects  using  a rubric  (Appendix 
1)  with  slight  modifications  from  an  established  rubric  by  Dodge  (2001).  Projects  were  assessed  in  six  areas  and 
received  a score  from  0 to  50  points.  Both  of  the  raters  were  former  instructors  of  this  course.  They  were 
knowledgeable  in  the  content  area  and  experienced  in  assessment.  Furthermore,  they  were  not  associated  with 
the  course  or  students  at  the  time  of  scoring,  which  minimized  any  potential  existing  biases.  Students  were 
instructed  to  remove  any  personal  information  in  their  projects.  Projects  from  both  experimental  and  control 
groups  were  coded  and  mixed  together.  Raters  could  not  identify  individual  students  or  identify  which  group 
projects  were  from. 

Inter-rater  reliability  was  assessed  for  the  two  raters.  The  Pearson  Correlation  between  the  scores  from 
two  raters  was  .680. 


Results 

Two  types  of  data  were  gathered  in  this  study.  The  first  type  compared  the  difference  of  student 
learning  represented  by  project  quality  between  the  experimental  and  control  groups.  The  second  type 
considered  students  general  perceptions  of  this  technology-mediated  peer  assessment  procedure. 

Difference  of  Projects  Quality 

Each  project  received  two  scores  from  two  independent  raters.  The  mean  score  was  calculated  and 
awarded  to  each  project.  ANOVA  was  utilized  to  test  if  there  was  any  significant  difference  between  the  project 
scores  of  the  control  group  and  the  experimental  group. 

Table  1 


Group 

Mean 

SD 

N 

Experimental 

37.95 

5.54 

26 

Control 

36.57 

7.28 

21 

The  difference  between  the  two  means  (37.95  vs.  36.57)  is  not  significant,  F (45)  = .545,  p = .464. 

Interactive  graph  (Figure  3)  shows  the  confidence  intervals  of  the  group  means  present  a large  overlap. 
There  is  a trend  that  the  mean  score  in  the  experimental  group  was  slightly  higher  than  that  of  the  control  group 
and  scores  in  the  experimental  group  were  located  in  a more  condensed  cluster. 
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Figure  3 


1=  experimental  group 


Students’  Perceptions  on  Peer  Assessment 

Twenty-two  students  in  the  experimental  group  responded  to  the  post-assessment  survey.  This  survey 
consisted  of  two  parts.  The  first  part  was  an  1 1-item  5-point  Likert  Scale  ranging  from  1 (strongly  disagree)  to  5 
(strongly  agree).  The  second  part  consisted  of  two  open-ended  questions  regarding  students’  likes  and  dislikes: 
“Please  specify  what  you  like  most  in  this  peer  assessment  procedure.”  “How  would  you  change  this  peer 
assessment  procedure?  And  why?” 

Table  2 


Peer  Assessment  Survey 


Mean 

SD 

Minimum 

Maximum 

1.1am  content  with  my  own  work. 

445 

.67 

3 

5 

2. 1 learn  more  from  peer  assessment  than 
from  traditional  teacher  assessment. 

3.64 

.73 

3 

5 

3.  The  procedures  on  how  to  do  peer 
assessment  are  clearly  outlined 

4.55 

.51 

4 

5 

4.  Peer  assessment  is  a worthwhile 
activity. 

4.18 

.59 

3 

5 

5.  Peers  have  adequate  knowledge  to 
evaluate  my  work. 

3.73 

.63 

3 

5 

6. 1 benefited  from  peers'  comments. 

4.32 

.65 

3 

5 

7.  The  peers'  comments  on  my  work 
were  fair. 

4.23 

.75 

3 

5 

8.  Peers  can  assess  fairly. 

3.95 

.65 

3 

5 

9. 1 have  benefited  from  marking  peers' 
work. 

3.91 

.92 

2 

5 

10  l took  a serious  attitude  towards 
marking  peers'  work. 

4.64 

49 

4 

5 

11  1 felt  that  1 was  critical  of  others 
when  marking  peers'  work 

4 00 

93 

2 

5 
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This  table  provides  a picture  of  students’  positive  perceptions  on  peer  assessment.  Students  reached  a 
general  satisfaction  level  for  all  of  the  items. 

For  the  first  open-ended  question  (“Please  specify  what  you  like  most  in  this  peer  assessment 
procedure.”),  three  major  themes  were  depicted.  First,  the  opportunity  to  review  and  grade  peers’  performance 
urged  students  on  to  greater  efforts  in  the  content  area  and  the  marking  criteria.  Secondly,  feedback  students 
received  from  peers  helped  them  improve  their  projects.  The  third  was  the  comfort  brought  by  anonymous 
marking  and  instant  feedback. 

For  the  second  open-ended  question  (“How  would  you  change  this  peer  assessment  procedure?  And 
why?”),  three  themes  emerged.  Several  students  stressed  their  satisfaction  with  this  technology -mediated 
process.  They  stated  that  they  wouldn’t  suggest  any  changes.  Some  students  would  have  liked  more  than  two 
peers  rating  their  projects.  They  found  it  difficult  to  decide  what  to  do  if  two  peers  gave  them  conflicting 
comments.  Some  students  asked  for  more  critical  and  constructive  feedback. 

Discussion 

This  study,  investigating  the  influence  of  peer  feedback  in  student  meaningful  learning  and  exploring 
student  satisfaction  level  of  this  process,  presented  us  an  interesting  picture.  Data  indicated  that  there  was  no 
significant  difference  of  the  project  quality  between  the  control  and  experimental  groups.  However,  post- 
assessment survey  revealed  students’  general  recognition  and  acceptance  of  this  process.  These  seemingly 
contradictory  outcomes  may  be  explained  in  part  by  the  following. 

First,  independent  ratings  were  used  to  compare  the  difference  of  students’  project  quality  between  two 
groups.  To  assure  the  reliability  and  consistency  of  scoring,  inter-rater  reliability  was  assessed.  Two 
independent  raters  graded  all  projects  according  to  the  rubric.  However,  their  grading  didn’t  reach  an  agreement 
at  a satisfactory  level  (the  Pearson  correlation  equaled  .680).  Therefore,  we  cannot  conclude  that  the  scoring 
was  reliable.  There  are  many  possibilities.  It  could  be  that  our  marking  criteria  was  not  categorized  and 
described  well  enough  for  raters  to  evaluate  students’  projects  and  reach  an  agreement.  Or  it  could  be  we  need 
to  provide  more  training  to  raters  before  they  started  grading.  Or  our  measurement  may  not  have  discriminated 
levels  of  quality. 

Secondly,  Topping  (1998)  suggested  that  the  benefits  from  peer  assessment  could  accumulate  anytime 
before,  during  or  after  the  procedure.  Peer  assessment  could  have  a positive  impact  on  students’  grades;  it  could 
also  aid  in  the  building  of  transferable  skills  and  the  foundation  of  lifelong  learning.  Like  most  peer  assessment 
studies  in  literature,  this  study  only  focused  on  summative  evaluation.  Though  statistically  it  revealed  no 
significant  difference  of  project  quality  between  the  control  and  experimental  groups,  the  general  agreement 
students  reached  in  the  post-assessment  survey  suggested  students  valued  peer  assessment  as  a worthwhile 
activity  and  they  benefited  marking  peers’  work.  If  formative  evaluation  was  applied,  there  might  be  some 
indicators  that  student  meaningful  learning  is  enhanced  by  this  process. 

Finally,  in  the  interactive  graph  (Figure  3),  though  the  confidence  levels  have  a large  overlap  and  the 
difference  between  the  two  groups  of  scores  is  not  significant,  there  is  a trend  that  the  mean  score  in  the 
experiment  groups  was  slightly  higher  than  that  of  the  control  groups.  If  a bigger  pool  of  students  had 
participated  in  this  study,  the  result  might  be  the  different.  The  variability  of  the  scores  in  the  experimental 
group  was  smaller  than  the  control  group.  Further  study  may  reveal  that  the  procedure  had  a differential  impact 
on  the  lower  scoring  students. 

Based  on  these  interpretations,  we  suggest  that  further  study  with  a larger  number  of  subjects  and  more 
instructors,  and  improved  quality  assessment  measures  is  warranted. 

The  merits  of  this  computer- mediated  peer  assessment  process — anonymity  and  promptness — were 
recognized  and  addressed  in  students’  survey  responses.  One  student  stated,  “it  helped  out  not  knowing  who  the 
person  was  critiquing  my  project”,  another  noted  “it  probably  puts  less  pressure  on  the  grader.”  Students  liked 
“the  instant  feedback”  from  peers.  At  the  same  time,  the  instructors  recognized  a significant  reduction  of 
management  workload.  All  the  data  were  automatically  summarized  by  the  system.  Students  and  instructors  had 
instant  access  to  data  once  they  were  generated.  This  certainly  reduced  the  administration  load. 

Though  the  difference  of  project  quality  between  groups  was  not  significant,  students  expressed  a 
rather  high  level  of  satisfaction  toward  this  computer-mediated  peer  assessment  process.  Overall,  we  felt  the 
peer  feedback  process  in  this  study  was  a worthwhile  activity.  During  this  process,  students  were  fully  engaged 
and  they  changed  their  roles  from  reviewers  to  reviewees,  and  then  improved  their  work.  During  this  process, 
students’  interaction  was  stimulated  and  their  critical  thinking  skills  were  fostered.  Compared  to  paper-based 
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systems,  a computer- mediated  system  is  certainly  promising  and  provides  advantages. 
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Appendix  1 WebQuest  Rubric 


Beginning  Developing  Accomplished  Score 

Overall  Aesthetics  (This  refers  to  the  WebQuest  page  itself,  not  the  external  resources  linked  to  it.) 

Overall  Visual  0 points  2 points  4 points 

Appeal 

There  are  few  or  Graphic  elements  Appropriate  and  thematic 

no  graphic  sometimes,  but  not  graphic  elements  are  used 

elements.  No  always,  contribute  to  make  visual  connections 

variation  in  layout  to  the  that  contribute  to  the 

or  typography.  understanding  of  understanding  of  concepts, 

concepts,  ideas  and  ideas  and  relationships. 

OR  relationships.  Differences  in  type  size 

There  is  some  and/or  color  are  used  well 

Color  is  garish  variation  in  type  and  consistently, 

and/or  typographic  size,  color,  and 
variations  are  layout, 

overused  and 
legibility  suffers. 

Background 
interferes  with  the 
readability. 

Navigation  & 0 points  2 points  4 points 

Flow 

Getting  through  There  are  a few  Navigation  is  seamless.  It 

the  lesson  is  places  where  the  is  always  clear  to  the 

confusing  and  learner  can  get  lost  learner  what  all  the  pieces 

unconventional.  and  not  know  are  and  how  to  get  to  them. 

Pages  can't  be  where  to  go  next, 

found  easily  and/or 
the  way  back  isn't 
clear. 

Mechanical  0 points  1 point  2 points 

Aspects 

There  are  more  There  are  some  No  mechanical  problems 

than  5 broken  broken  links,  noted. 

links,  misplaced  or  misplaced  or 

missing  images,  missing  images, 

badly  sized  tables,  badly  sized  tables, 

misspellings  misspellings 

and/or  and/or 

grammatical  grammatical 

errors.  errors. 

Introduction 

Motivational  0 points  1 point  2 points 

Effectiveness  of 

Introduction  The  introduction  is  The  introduction  The  introduction  draws  the 

purely  factual,  relates  somewhat  reader  into  the  lesson  by 

with  no  appeal  to  to  the  learner's  relating  to  the  learner's 

relevance  or  social  interests  and/or  interests  or  goals  and/or 

importance  describes  a engagingly  describing  a 

compelling compelling  question  or 
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OR  question  or  problem, 

problem. 

The  scenario  posed 
is  transparently 
bogus  and  doesn't 
respect  the  media 
literacy  of  today's 
learners. 

Cognitive  0 points  1 point  2 points 

Effectiveness  of 

the  Introduction  The  introduction  The  introduction  The  introduction  builds  on 

doesn't  prepare  the  makes  some  learner's  prior  knowledge 

reader  for  what  is  reference  to  or  effectively  prepares  the 

to  come,  or  build  learner's  prior  learner  by  foreshadowing 

on  what  the  learner  knowledge  or  what  the  lesson  is  about, 

already  knows.  previews  to  some 
extent  what  the 
lesson  is  about. 

Task  (The  task  is  the  end  result  of  student  efforts...  not  the  steps  involved  in  getting  there.) 

Connection  of  0 points  2 point  4 points 

Task  to 

Standards  The  task  is  not  The  task  is  The  task  is  referenced  to 

related  to  referenced  to  standards  and  is  clearly 

standards.  standards  but  is  not  connected  to  what  students 

clearly  connected  must  know  and  be  able  to 
to  what  students  do  to  achieve  proficiency 
must  know  and  be  of  those  standards, 
able  to  do  to 
achieve 
proficiency  of 
those  standards. 

Cognitive  Level  0 points  3 points  6 points 

of  the  Task 

Task  requires  Task  is  doable  but  Task  is  doable  and 

simply  is  limited  in  its  engaging,  and  elicits 

comprehending  or  significance  to  thinking  that  goes  beyond 

retelling  of  students'  lives.  The  rote  comprehension.  The 

information  found  task  requires  task  requires  synthesis  of 

on  web  pages  and  analysis  of  multiple  sources  of 

answering  factual  information  and/or  information,  and/or  taking 

questions.  putting  together  a position,  and/or  going 

information  from  beyond  the  data  given  and 
several  sources.  making  a generalization  or 
creative  product. 

Process  (The  process  is  the  step-by-step  description  of  how  students  will  accomplish  the  task. ) 

Clarity  of  o points  2 points  4 points 

Process 

Process  is  not  Some  directions  Every  step  is  clearly  stated, 

clearly  stated.  are  given,  but  there  Most  students  would  know 

Students  would  not  is  missing  exactly  where  they  are  at 

know  exactly  what  information.  each  step  of  the  process 

they  were Students  might  be  and  know  what  to  do  next. 
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supposed  to  do  just  confused, 
from  reading  this. 

Scaffolding  of  0 points  3 points  6 points  II 

Process 

The  process  lacks  Strategies  and  The  process  provides 

strategies  and  organizational  students  coming  in  at 

organizational  tools  embedded  in  different  entry  levels  with 

tools  needed  for  the  process  are  strategies  and 

students  to  gain  the  insufficient  to  organizational  tools  to 

knowledge  needed  ensure  that  all  access  and  gain  the 

to  complete  the  students  will  gain  knowledge  needed  to 

task.  the  knowledge  complete  the  task, 

needed  to  complete 

Activities  are  of  the  task.  Activities  are  clearly 

little  significance  related  and  designed  to 

to  one  another  Some  of  the  take  the  students  from 

and/or  to  the  activities  do  not  basic  knowledge  to  higher 

accomplishment  of  relate  specifically  level  thinking, 

the  task.  to  the 

accomplishment  of 
the  task. 

Richness  of  0 points  1 points  2 points 

Process 

Few  steps,  no  Some  separate  Different  roles  are  assigned 

separate  roles  tasks  or  roles  to  help  students  understand 

assigned.  assigned.  More  different  perspectives 

complex  activities  and/or  share  responsibility 

required.  in  accomplishing  the  task. 

Resources  (Note:  you  should  evaluate  all  resources  linked  to  the  page,  even  if  they  are  in  sections  other 
than  the  Process  block.  Also  note  that  books,  video  and  other  off-line  resources  can  and  should  be  used 

where  appropriate.) 

Relevance  & 0 points  2 point  4 points 

Quantity  of 

Resources  Resources  There  is  some  There  is  a clear  and 

provided  are  not  connection  meaningful  connection 

sufficient  for  between  the  between  all  the  resources 

students  to  resources  and  the  and  the  information  needed 

accomplish  the  information  for  students  to  accomplish 

task.  needed  for  the  task.  Every  resource 

students  to  carries  its  weight. 

OR  accomplish  the 

task.  Some 

There  are  too  resources  don't  add 

many  resources  for  anything  new. 
learners  to  look  at 
in  a reasonable 
time. 

Quality  of  0 points  2 points  4 points 

Resources 

Links  are  Some  links  carry  Links  make  excellent  use 

mundane.  They  information  not  of  the  Web's  timeliness  and 

lead  to  information  ordinarily  found  in  colorfulness. 
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that  could  be  found 
in  a classroom 
encyclopedia. 

a classroom. 

Varied  resources  provide 
enough  meaningful 
information  for  students  to 
think  deeply. 

Evaluation 

Clarity  of 

Evaluation 

Criteria 

0 points 

Criteria  for  success 
are  not  described. 

3 points 

Criteria  for  success 
are  at  least 
partially  described. 

6 points 

Criteria  for  success  are 
clearly  stated  in  the  form  of 
a rubric.  Criteria  include 
qualitative  as  well  as 
quantitative  descriptors. 

The  evaluation  instrument 
clearly  measures  what 
students  must  know  and  be 
able  to  do  to  accomplish 
the  task. 

Tol 

tal  Score 

/50 
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Abstract 

This  study  articulates  the  construct  of  social  presence  and  develops  a social  presence  questionnaire  for 
examining  online  collaborative  learning  with  tests  for  reliability  and  validity.  Questionnaire  items  were 
developed  by  revising  the  social  presence  questionnaire  developed  by  Picciano  in  2002  as  well  as  reviewing 
research  in  the  literature  of  computer  support  for  cooperative  systems  ( CSCW).  Twenty  items  were  developed 
and  administered  to  15  graduate  students  taking  an  online  course.  Exploratory  factor  and  reliability  analyses 
resulted  in  the  identification  of  12  items  reflecting  online  social  presence. 

Introduction 

Computer  mediated  communication  (CMC)  is  a substantial  aspect  of  learning  at  a distance,  and  Short 
et  al.  (1976)  claim  that  social  presence  is  the  critical  factor  in  a communication  medium.  Social  presence  is 
defined  as  “the  ability  of  participants  in  a community  of  inquiry  to  project  themselves  socially  and  emotionally, 
as  real  people  through  the  medium  of  communication  being  used”  (Garrison  & Anderson,  2003).  Online 
learning  environments  which  feature  mainly  asynchronous  text -based  CMC  have  been  criticized  for  their  lack 
of  support  for  social  presence,  and  this  lack  of  support  for  social  presence  may  impact  the  sense  of  belonging 
and  acceptance  in  a group  (Rovai,  2002). 

In  the  earliest  research  of  social  presence.  Short  et  al.  (1976)  related  the  concepts  of  intimacy  and 
immediacy  with  social  presence.  This  early  work  suggested  that  intimacy  and  immediacy  enhance  social 
presence  (Gunawardena,  1995).  Social  presence  also  was  defined  as  a quality  of  the  medium  itself.  They  used 
the  semantic  differential  technique  with  bipolar  scales  to  assess  social  presence  in  face  to  face  television  and 
audio  systems  around  four  dimensions:  unsociable -sociable,  insensitive-sensitive,  impersonal-personal,  and 
cold-warm.  Since  an  asynchronous  text -based  CMC  has  different  attributes  from  one-way  television, 
Gunawardena  (1995)  examine  social  presence  as  an  attribute  of  a computer  conference  by  revising  Short  et  al.’s 
scales  to  17  5 -point  bipolar  scales  that  characterized  the  intimacy  of  the  medium. 

Further,  Gunawardena  & Zittle  (1997)  developed  a social  presence  scale  containing  14  questionnaire 
items  that  embodied  the  concept  of  immediacy  to  focus  on  perceived  sense  of  online  community  and  degree  of 
social  comfort  with  CMC.  Tu  (2002)  argued  that  current  social  presence  instruments  are  unable  to  capture  a 
thorough  perception  of  social  presence  and  asserted  that  social  presence  is  a complicated  construct  containing  4 
dimensions:  social  context,  online  communication,  interactivity  and  privacy.  His  social  presence  and  privacy 
questionnaire  instrument  measures  social  presence  in  email,  bulletin  board  and  real-time  discussions  and 
contains  17  social  presence  items  and  13  privacy  items  with  a five  point  likert  scale  and  12  demographic 
responses.  In  addition,  Rourke  et  al.  (2001)  directly  examined  responses  of  computer  conference  participants 
through  content  analysis  of  conferencing  transcripts  and  developed  three  categories  and  indicators  to  assess 
social  presence  including  affective  responses,  interactive  responses,  and  group  cohesive  responses. 

Our  review  of  the  social  presence  literature  and  instrumentation  suggests  there  is  still  a lack  of 
agreement  about  how  to  conceptualize  and  measure  social  presence,  but  that  there  is  also  a growing 
appreciation  for  its  potential  to  explain  participation  and  outcomes  in  distance  learning.  No  reliability  and 
validity  assessments  of  the  social  presence  instruments  developed  by  Short  et  al.  in  1976  & Gunawardena  in 
1995  were  reported.  For  the  instrument  developed  by  Gunawardena  & Zittle  in  1997,  concurrent  validity  of  the 
social  presence  scales  was  indicated  by  the  strong  and  positive  correlation  with  bipolar  social  indicators  based 
on  Short  et  al.’s  instrument;  however,  the  scale  itself  was  not  validated.  The  social  presence  instrument 
developed  by  Tu  in  2002  has  been  validated;  however,  his  instrument  mainly  focused  on  participants’  attitudes 
toward  CMC  in  a general  context.  It  is  unclear  whether  the  reported  relationship  between  attitude  toward  CMC 
and  the  experience  of  social  presence  would  hold  when  confronted  with  specific  tasks  or  opportunities  in 
specific  social  groupings. 

Previous  research  has  examined  the  association  of  social  presence  with  participation  and  outcomes  in 
distance  learning.  Social  presence  has  been  associated  with  enhanced  online  social  interaction  (Tu  & Mclsaac, 
2002).  Social  presence  is  also  seen  to  influence  not  only  online  activities  generally  designated  as  group  projects, 
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but  also  those  usually  designated  as  individual  projects  (Richardson  &Swan,  2003).  In  addition,  students  with 
high  overall  perceptions  of  social  presence  scored  high  in  terms  of  perceived  learning  and  perceived  satisfaction 
with  the  instructor  (Richardson  &Swan,  2003).  Students  with  high  overall  perceptions  of  social  presence  are 
also  most  likely  to  enhance  their  socio -emotional  experience  by  adopting  different  ways  to  express  their  affect 
in  an  asynchronous  text -based  learning  environment  (Gunwardena  & Zittle,  1997).  Finally  research  has  also 
shown  that  instructors  or  moderators  of  online  communities  can  cultivate  social  presence  by  developing 
interaction  skills  that  create  a sense  of  social  presence  (Gunawardena,  1995). 

The  purpose  of  the  work  presented  in  this  paper  is  to  further  articulate  the  construct  of  social  presence 
and  to  develop  a social  presence  instrument  which  can  be  used  to  examine  social  presence  in  online 
collaborative  learning. 


Questionnaire  Development 

To  measure  the  social  presence  of  students  working  collaboratively  in  an  online  course,  two  strategies 
were  used  to  develop  items  for  the  social  presence  questionnaire.  First,  the  first  10  items  were  developed  by 
surveying  social  presence  literatures  and  adapting  items  from  the  social  presence  questionnaire  developed  by 
Picciano  (2002)  based  on  a questionnaire  developed  by  Tu  (2001).  Second,  the  last  10  items  were  developed 
newly  from  our  reading  in  the  literature  of  computer  support  for  cooperative  systems  (CSCW).  In  this  literature 
there  is  a greater  emphasis  on  social  navigation  and  awareness  of  others  than  we  have  found  in  CMC  more 
generally  and  especially  distance  learning.  This  literature  emphasizes  the  role  that  awareness  of  the  actions  of 
others  and  the  understanding  that  others  are  aware  of  your  actions  shapes  action.  See  Munro,  et  ah,  (1999)  and 
Hook,  et  ah,  (2003)  for  good  compilations  of  this  research.  In  all,  20  statements  were  created  to  measure  social 
presence.  Statements  were  placed  on  a 7 -point  continuum  with  endpoints  of  strongly  agree  (1)  to  strongly 
disagree  (7). 


Method 

In  a pilot  study  the  20  items  were  administered  to  15  graduate  students  in  an  online  graduate  level 
course  delivered  through  Shadow  netWorkspace™  (SNS)  during  the  fall  of  2003.  SNS  is  open  source  software 
using  the  GNU  General  Public  License  (GPL).  The  software  can  be  freely  downloaded  and  distributed  under  the 
terms  of  the  GPL.  Shadow  netWorkspace  is  freely  available  to  anyone  at  http://sns.internetschools.org.  The 
online  course  was  organized  into  8 weekly  group  activities  and  two  individual  projects.  After  the  third  weekly 
activity,  all  the  students  were  asked  to  complete  the  web-based  social  presence  questionnaire. 

Exploratory  Factor  Analysis  and  Reliability  Analysis 

Responses  to  the  20  items  were  subjected  to  exploratory  factor  analysis  using  principle  component 
with  varimax  rotation.  Initial  factor  analysis  procedure  showed  the  20  item  questionnaire  to  have  three  factors 
having  eigenvalues  of  8.114,  3.277,  and  2.329  that  accounted  for  40.570%,  16.384%,  and  11.643%  of  the 
variance,  respectively.  The  factor  analysis  was  repeated  and  6 items  were  deleted  since  they  were  found  to 
contribute  approximately  equally  into  at  least  two  factors.  Finally,  the  three  factors  which  explained  73.889%  of 
total  variance  were  named  as  “perception  of  the  assistance  of  group  activity  to  learning”,  “social  comfort  of 
expressing  and  sensing  affect”,  and  “social  navigation”.  Cronbach’s  coefficient  alpha  was  computed  for  the  14 

Table  1.  Factor  loadings  for  social  presence  questionnaire 


Item  Factor  loading 

Factor  1 : Perception  of  the  assistance  of  group  activity  to  learning 


1 

I felt  like  I was  a member  of  a group  during  this  past  week  activities 
I felt  comfortable  participating  in  this  past  week  online  group 

0.753 

2 

activities. 

0.811 

8 

I felt  I came  to  know  the  other  students  in  this  past  week  online  group 
activities 

0.746 

16 

This  past  week  online  group  activities  helped  me  accomplish  the 
assignment  with  higher  quality  than  if  I were  working  alone. 

0.804 

17 

This  past  week  online  group  activities  helped  me  learn  more 
efficiently  than  if  I were  working  alone. 

0.831 
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Factor  2:  Social  comfort  of  expressing  and  sensing  affect 


4 

I felt  comfortable  expressing  my  feelings  during  this  past  week 
activities. 

0.833 

6 

I felt  comfortable  expressing  my  humor. 

0.918 

7 

I was  able  to  appreciate  the  humor  of  members  of  the  group. 

0.867 

9 

I was  able  to  form  distinct  individual  impressions  of  some  group 
members  during  the  online  group  activities. 

0.833 

Factor  3:  Social  navigation 

14 

Actions  by  other  members  of  my  group  usually  influenced  me  to  do 
further  work. 

0.891 

15 

Knowing  that  other  members  of  my  group  were  aware  of  my  work 
influenced  the  frequency  and/or  quality  of  my  work. 

0.690 

19 

Knowing  what  other  members  of  the  group  did  helped  me  know  what 
to  do. 

0.792 

item  social  presence  questionnaire  as  a test  of  internal  consistency.  Cronbach’s  coefficient  alpha  values  for  three 
factors  were  0.6747,  0.6649,  and  0.7031  respectively.  By  dropping  one  item  individually  from  “perception  of 
the  assistance  of  group  activity  to  learning’'  factor  and  “social  comfort  of  expressing  and  sensing  affect"  factor, 
their  reliabilities  were  raised  to  0.8905  and  0.9218  respectively.  Alpha  of  the  entire  questionnaire  of  12  items 
was  0.8402  (M=55.6667,  SD=  11.8181).  The  scale’s  reliability  met  acceptable  standards  of  0.70  and  above  and 
can  be  interpreted  as  internally  consistent  or  as  measuring  the  same  phenomenon  (Bowers  & Courtright,  1984). 
Table  1 shows  factor  loadings  of  three  factors  for  12  items. 

Conclusion 

Our  work  is  leading  to  a social  presence  questionnaire  which  can  be  used  in  online  collaborative 
learning.  The  exploratory  factor  analysis  isolated  three  factors:  “perception  of  the  assistance  of  group  activity  to 
learning",  “social  comfort  of  expressing  and  sensing  affect”,  and  “social  navigation”.  In  this  pilot  study,  the 
sample  size  for  factor  analysis  was  far  from  an  appropriate  sample  size  of  close  to  300  cases  (Tabachnick  & 
Fidell,  2001);  hence,  future  studies  are  planned  with  larger  sample  sizes. 
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Abstract 

The  deployment  of  immersive,  non-re  strictive  environments  for  instruction  and  learning  presents  a new 
set  of  challenges  for  instructional  designers  and  educators.  Adopting  the  conceptual  frameworks  ofSherin  ’s 
(2002)  learning  while  teaching  and  Vygotsky’s  (1978)  cultural  development  via  the  mediation  of  tools,  this 
paper  explores  one  professor's  pedagogical  approaches  used  for  instructing  fluid  dynamics  concepts  within  the 
Metaverse,  a highly  innovative,  physically  immersive  visualization  display  that  does  not  require  the  use  of  head- 
mounted  or  other  restrictive  devices. 


Introduction 

The  emergence  of  virtual  realities  (VR)  in  the  educational  technology  landscape  in  recent  years  has 
aroused  the  enthusiasm  of  instructional  designers  and  educators,  especially  among  instructors  in  the  discipline 
of  science.  Virtual  reality  technologies  enable  a new  dimension  of  representing  difficult  scientific  concepts  and 
processes  that  2-D  desktop  graphical  representations  have  not  been  able  to  offer  (Jimoyiannis  & Komis,  2001; 
Fiolhais  & Trindale,  1998).  However,  integrating  these  burgeoning  technologies  into  school  curriculum  and 
higher  education  presents  a new  set  of  challenges  within  the  realms  of  learning  and  instruction  (Barajas  & 

Owen,  2000).  In  the  past  decade,  a plethora  of  discussions  and  inquiries  in  educational  technology  journals  and 
conference  proceedings  have  placed  significant  emphasis  on  various  aspects  of  simulated  desktop  virtual 
environments,  2-D  multimedia  representations  and  web-based  instruction  and  learning  (Davies,  2002; 

Antonietti  & Cantoia,  2000;  Maher  & Corbit,  2002;  Gabrielli,  S.,  Rogers,  Y.,  & Scaife,  M.  2000;  Gazit,  E.,  & 
Chen,  D.,  2003).  Moreover,  empirical  studies  on  the  applications  and  implementations  of  immersive,  non- 
desktop virtual  reality  interfaces  in  education,  especially  scalable  virtual  reality  infrastructures  are  scant 
(Crosier,  Cobb,  & Wilson,  2002). 

To  this  end,  the  Metaverse,  a NSF-funded  interdisciplinary  research  project  at  the  University  of 
Kentucky,  has  marked  a milestone  in  the  development  of  educational  technologies.  In  addition  to  developing 
self-configuring,  scalable  visual  displays,  the  Metaverse  makes  it  possible  for  learning  and  instruction  in  a 
commodity  hardware  supported  immersive,  non-restrictive  environment.  The  objectives  are  to  support 
meaningful  human-computer  interaction  and  to  understand  the  impact  these  visual  environments  have  on 
education.  For  the  past  three  years,  the  Metaverse  team  has  been  translating,  designing  and  implementing 
prototype  instructional  materials  into  immersive  displays.  One  of  the  preeminent  tasks  that  the  educators  of  the 
team  face  is  to  seek  ways  of  facilitating  instruction  in  this  dynamic,  non-linear  teaching  and  learning 
environment.  Many  initial  questions  arise.  Chief  among  them,  for  example,  is  how  can  teachers  learn  to  use  an 
immersive  environment  in  which  developing  their  own  understanding  of  the  nature  of  the  new  computer- 
generated 3-D  environment  is  of  primal  concern?  As  McLellan  (2003)  pointed  out,  careful  and  continual 
examination  of  virtual  reality  technologies,  especially  within  the  context  of  learning  and  instruction  are 
important  for  two  reasons:  1)  virtual  reality  technologies  are  still  in  their  nascent  stage  of  development  as  a 
category  in  educational  technologies,  and  2)  rapid  technological  improvement  posts  concerns  for  the  issue  of 
outdated  technological  capabilities. 

Thus,  the  purpose  of  this  exploratory  study  was  to  analyze  the  emerging  pedagogical  strategies  within 
a non-restrictive,  immersive  environment.  More  specifically,  this  study  examined  the  teaching  approaches 
employed  by  a professor  in  the  instruction  of  fluid  dynamics  concepts  using  in  the  Metaverse  display.  Using 
Sherin’s  (2002)  concept  of  “teaching  while  learning’'  and  Vygotsky’s  (1978)  notion  of  “cultural  development” 
through  the  mediation  of  tools,  this  paper  seeks  to  understand  the  instructor’s  integration  of  subject  matter 
knowledge  and  his  newly  acquired  knowledge  in  deploying  the  immersive  technology.  The  goals  are  to:  a)  shed 
some  light  on  the  development  of  effective  instruction,  b)  to  optimize  the  full  potential  that  this  inherently 
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“open-ended”  learning  environment  affords,  and  c)  to  provide  an  initial  step  toward  future  studies  of  scalable, 
immersive,  virtual  learning  environments  for  educational  applications. 


Conceptual  Framework  for  the  Study 

According  to  Sherin  (2002),  teachers  go  through  a process  of  transforming,  adapting,  and  negotiating 
when  faced  with  changes  in  instructional  curriculum  or  materials.  “These  classes  of  interactions  between  the 
teachers’  content  knowledge  and  the  implementation  of  the  novel  curriculum.  . .represent  the  different  ways 
that  teachers  apply  their  content  knowledge  as  they  attempt  to  use  new  materials”  (Sherin,  2002,  p.  129). 
Moreover,  Sherin  (2002)  posits  that  it  is  only  through  the  process  of  negotiating  with  new  instructional 
materials  or  situations  that  teachers  truly  engage  in  the  active  role  of  learning.  In  other  words,  teachers  who 
work  with  new  instructional  content  often  experience  a series  of  changes  that  prompt  them  to  arrive  at  an 
innovative  stage  of  developing  new  ways  of  interpreting  and  presenting  their  newly  acquired  content 
knowledge.  However,  it  is  inadequate  to  address  the  wide  spectrum  of  pedagogical  strategies  required  to 
deliver  meaningful  instruction  within  an  immersive,  non-restrictive  environment  such  as  the  Metaverse  display. 
Specifically,  additional  concepts  are  needed  to  frame  the  interaction  between  the  instructor  and  the  immediate 
environment. 

Drawing  upon  Vygotsky’s  (1978)  concept  of  “cultural  development”,  the  authors  of  this  study  argue 
that  an  immersive  virtual  environment  can  be  viewed  as  a “culture”  in  which  both  the  instructors  and  students 
alike  develop  their  own  tools  and  verbal  and  nonverbal  signs  in  the  process  of  becoming  familiar  and  efficient 
with  their  navigation  within  the  environment.  According  to  Vygotsky  (1978),  tools  and  signs  represent 
different  ways  of  orienting  human  behavior  and  are  the  essence  of  mediated  activities. 

The  tools’  function  is  to  serve  as  the  conductor  of  human  influence  on  the  object  of  activity,  it 
is  externally  oriented;  it  must  lead  to  changes  in  objects.  It  is  a means  by  which  human 
external  activity  is  aimed  at  mastering,  and  triumphing  over,  nature.  The  sign,  on  the  other 
hand,  changes  nothing  in  the  object  of  a psychological  operation.  It  is  a means  of  internal 
activity  aimed  at  mastering  oneself;  the  sign  is  internally  oriented  (Vygotsky,  1978,  p.  55). 

Within  the  context  of  this  study,  tools,  referred  to  by  Vygotsky  (1978)  as  the  external  object,  imply  the  user’s 
physical  body  movement  to  navigate  in  the  immersive  visual  display.  Analyzing  the  body  movement  that 
revolves  around  a series  of  actions,  described  by  Vygotsky  (1978)  as  “true  gesture”,  the  authors  attempt  to 
examine  the  embedded  dimensions  of  teaching  that  may  be  unique  to  immersive  environments. 

The  Environment 

This  exploratory  observational  case  study  is  anchored  in  the  Metaverse,  a non-restrictive,  immersive 
environment.  One  of  the  fundamental  concepts  behind  this  infrastructure  is  its  visually  immersive  and 
interactive  affordances.  The  500  square -foot  room  with  its  fourteen  ceiling-mounted  projectors  and  calibrating 
systems  enable  a single  user  to  move  around  freely  without  constrain  of  devices  like  the  head-mounted  display, 
data  gloves,  or  stereo  glasses,  which  are  characteristics  of  typical  CAVE-like  immersive  environments.  As  the 
user  moves  to  various  locations  in  the  projected  two -wall  and  floor  display  environment  of  the  Metaverse,  a 
head-tracking  device  on  a fedora  hat  or  being  held  by  the  user  corrects  for  changes  in  the  display  in  real-time. 

In  another  words,  the  viewpoint  or  perspective  changes  simultaneously  as  the  user  moves  within  the 
environment.  Further,  a user  in  the  display  can  obtain  top,  bottom,  interior  and  exterior  views  of  the  display  or 
the  projected  image  by  strategically  positioning  him  or  her  in  the  environment.  The  problem  in  using  this  type 
of  physically  immersed  environment  surfaces  when  the  cues  of  depth  in  virtual  environments  are  often  different 
from  that  of  the  natural  world  (Wann  & Mon-Williams,  1996).  In  addition,  time  lag  in  displaying  the 
corresponding  view  as  to  the  user’s  body  movement  also  creates  challenging  usability /context  -for-learning 
issues. 


Subject  Matter  Content 

The  subject  matter  content  of  the  inquiry  pertained  to  the  behaviors  of  a laboratory  simulated  fire 
whirl.  “Fire  whirls  occur  infrequently,  usually  as  the  result  of  . . . large  scale  wildland  or  urban  fires — the 
former  usually  caused  by  lightning  strikes,  and  the  latter  often  due  to  earthquakes  or  some  similar  disaster” 
(McDonough  & Loh,  2003.  p.  1).  Studies  on  the  characteristics  of  fire  whirls  have  been  sparse  despite  the 
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potential  damages  that  these  phenomena  can  cause  and  the  possible  control  measures  of  small  pool  fire  in 
industrial  applications.  Selection  of  this  subject  matter  content  also  relates  to  the  availability  and  accessibility 
of  the  visual  display  for  the  Metaverse  environment. 

As  the  technical  development  of  the  Metaverse  and  its  contents  is  an  ongoing  process,  the  simulation 
of  the  fire  whirl  in  the  Metaverse  display  during  this  study  was  a simple  cylinder-shape  projected  image  with 
areas  of  grey,  blue,  orange  and  yellow.  The  concept  of  this  display  is  to  show  the  circular  moving  flow  field 
and  the  color-coded  regions  of  flame  temperature.  Walking  inside  or  to  other  regions  can  lead  to  changes  in  the 
appearance  of  the  flame  and  the  temperature.  One  of  the  objectives  in  creating  this  simulation  in  the  Metaverse 
was  to  provide  visualization  of  the  turbulent  flow  behaviors  of  the  flame  and  its  various  burning  regions. 

The  Participant 

Selection  of  the  professor  for  the  study  was  based  upon  his  involvement  with  the  NSF  grant  of  the 
Metaverse  project,  his  expertise  in  the  field  of  fluid  dynamics,  and  his  enthusiasm  in  experimenting  with  the 
immersive  visualization  interfaces.  The  primary  task  of  the  professor  was  to  present  and  demonstrate  the 
characteristics  and  turbulent  flow  behaviors  of  the  simulated  fire  whirl  described  earlier.  Yin  (1994)  has  noted 
the  acceptability  of  a case  study  of  one  to  describe  or  explore  a unique  or  extreme  case.  The  unique  situation 
criterion  is  clearly  met  in  this  instance. 

Data  Collection  Procedures 

To  maximize  the  trustworthiness  and  reliability  of  the  study,  the  authors  adopted  various  data 
collection  resources . For  instance,  the  eight -minute  presentation  of  the  fire  whirl  in  the  Metaverse  display  was 
digitally  video  and  audio  recorded.  The  video  recording  was  then  transferred  to  CD -Rom,  and  the  content  was 
transcribed  verbatim  and  member  checking  by  the  professor.  Moreover,  other  methods  of  collecting  data 
included  a focused  interview  with  the  professor  after  examining  the  video-recorded  presentation,  email 
communication,  and  field  notes.  To  compare  the  behaviors  of  the  professor  in  the  immersive  display  with  his 
in-class  teaching  methods,  the  lead  author  conducted  eight  fifty-minute  classroom  observations  for  two 
semesters  prior  to  analyzing  the  recorded  video  and  transcript. 

Analysis 

The  authors  based  the  analytic  techniques  of  this  case  study  on  both  the  conceptual  frameworks  of 
Sherin’s  (2002)  learning  while  teaching  and  Vygotsky’s  (1978)  cultural  development  through  the  mediation  of 
tools.  As  such,  the  researchers  examined  the  interaction  categories  of  transform,  adapt,  and  negotiate,  and  the 
mediation  of  tool  (body  movement)  in  two  phrases:  1)  identified,  coded,  and  analyzed  the  incidences  that 
negotiation  took  place  in  terms  of  the  subject  matter  content,  and  2)  identified,  coded,  and  analyzed  the 
incidences  of  tool  (the  body)  manipulation.  These  analytic  strategies  followed  the  case  study  tradition  of 
pattern -matching  (Yin,  1994),  the  coding  and  categorization  of  evidence  (Stakes,  1995),  the  displaying  of  events 
in  table  and  form,  and  the  tabulation  of  the  frequency  of  incidences  (Miles  and  Huberman,  1984). 

Aside  from  analyzing  the  video  and  its  transcript,  the  authors  also  conducted  an  ongoing  process  of 
triangulating  field  notes  and  transcribed  interview  to  search  for  emerging  patterns  and  themes.  Stake  (1995)  has 
posited  that  triangulation  increases  the  validity  of  the  study  and  thus  ensures  the  accuracy  in  interpreting  the 
collected  data.  Moreover,  Yin  (1994)  and  Stake  (1995)  maintain  that  collecting  and  analyzing  varied  evidence 
sources  such  as  interviews,  observation,  and  documents  verifies  the  authenticity  of  the  phenomena  under 
investigation. 


Results  and  Discussion 

Analyzing  the  transcript  and  taped  video  reveal  several  findings  regarding  the  professor’s  attempts  to 
cope  with  delivering  his  content  knowledge  in  the  fire  whirl  immersive  display  (new  context).  Table  1 
provides  samples  of  incidences  that  illustrate  the  instructor’s  use  of  existing  content  knowledge  to  implement  a 
new  context  through  the  process  of  transformation.  Furthermore,  the  table  shows  evidence  of  the  professor’s 
skill  in  developing  new  content  knowledge  and  implementing  the  information  into  the  presentation  (process  of 
adaptation),  and  his  approach  in  developing  new  content  knowledge  and  modifying  the  content  information  as 
he  proceeded  with  his  instruction  (process  of  negotiation).  Although  each  of  these  three  classes  of  interactions 
reflects  learning  through  teaching,  Sherin  (2002)  maintains  that  attention  should  be  placed  on  negotiation  as  it 
signifies  the  teachers’  active  role  of  learning  during  instruction.  By  definition,  negotiation  occurs  when 
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“teachers  develop  new  content  knowledge  and  at  the  same  time  make  changes  in  a lesson  as  it  unfolds  in  the 
classroom.  . . In  other  words,  the  teacher  not  only  develops  new  content  knowledge  but  also  uses  this 
knowledge  to  interpret  the  lesson  in  progress  and  decide  how  to  proceed”  (p.  130). 

Throughout  the  presentation,  the  professor  negotiated  his  fire  whirl  content  and  the  immersive 
environment  by  making  comments  that  centered  on  his  justification  of  what  could  have  happened  if  the 
immersive  display  was  programmed  with  a complete  data  set.  For  instance,  in  the  beginning  of  the 
presentation,  he  remarked,  “If  we  had  a complete  simulation,  we  will  be  able  not  only  to  see  the  temperature  in 
this  thing,  but  we  will  be  able  to  see  the  flow  field  down  underneath.”  In  another  incidence,  the  professor 
developed  and  integrated  his  new  content  knowledge  within  the  context  of  the  environment  by  pointing  out  the 
advantage  and  disadvantage  of  having  the  immersive  visual  display.  “Now,  one  of  the  things  that’s  missing 
from  the  system  that  we  will  eventually  have  in  place  is  the  ability  to  know  just  exactly  where  we  are  in  a given 
time.  As  we  stand  right  now,  I don’t  know  exactly  where  I am.  But,  by  walking  around  in  here  at  the  very  least 
we  can  see  how  the  different  regions  connect  up  to  the  other.”  These  sample  incidences  illustrate  the 
professor’s  creativity  in  generating  new  content  about  the  flame  as  he  explained  its  turbulent  behaviors  in  the 
immersive  display.  Further,  a comparison  of  the  professor’s  frequent  integration  of  new  information  during  the 
presentation  to  the  highly  structured,  non -interactive  classroom  lectures  also  suggests  the  instructor’s  active 
negotiation  within  the  immersive  environment. 

Placing  Vygotsky’s  (1978)  concept  of  “cultural  development”  through  the  mediation  of  tools  in  the 
context  of  this  study,  the  instructor  demonstrated  his  role  in  creating  a culture  for  the  environment  and  his 
development  in  gaining  competency  in  deploying  the  immersive  display.  By  moving  or  situating  his  body  (the 
tool)  in  various  positions  and  locations,  the  professor  was  able  to  obtain  different  perspectives  or  views  of  the 
fire  whirl  visual 

representation  (Figure  1,  2,  & 3).  Moreover,  competencies  in  obtaining  the  desired  views  in  the  display 
required  skills  through  repeated  actions.  For  instance,  the  following  statement  made  by  the  professor  illustrated 
his  lack  of  confidence  in  obtaining  the  desired  view  of  the  display  and  his  challenge  in  mastering  his  tool  (body 
movement).  “If  we  get  in  the  right  place  here,  we  maybe  get  to  a place  where  I get  fire  all  around  me.  It’s 
moving  a little  erratically.  Let’s  try  this  way.  . . there  we  go.”  As  the  instructor  gained  knowledge  with 
repeated  use  of  the  fire  whirl  immersive  display,  his  accuracy  in  obtaining  the  desired  view  of  the  display  with 
the  least  amount  of  time  increased.  “The  first  thing  we  are  going  to  do  is  to  see  if  we  can  look  down  on  top  of 
this  to  see  some  evidence  of  burning.  And  we  can  see  that  here.  Over  here.  . . and  it’s  stabilized.  And  now  we 
bring  the  tracker  back  down  and  actually  get  underneath  it  and  look  at  it  from  the  bottom”.  This  instance,  which 
occurred  toward  the  end  of  the  presentation,  exemplified  the  professor’s  skills  in  mastering  the  nature  (the 
immersive  environment)  and  mastering  of  behaviors.  He  became  more  decisive  in  manipulating  his  body 
movement  and  in  taking  the  required  actions 


Interaction 

Sample  Evidence  with  number  of  similar  instances 

Content 

Novel 

Class 

Knowledge 

Lesson 

Transform 

Discussed  the  behaviors  of  flow  field  by  interacting  with  and  showing 
the  visual  display  (4  incidences) 

Unchanged 

Changed 

Adapt 

Invented  explanation  for  the  different  color  regions  and  heat 
temperature  when  asked  about  the  indicators  of  the  changes  in  time  in 
the  immersive  display  (2  incidences) 

Changed 

Unchanged 

Negotiate 

Statement  such  as  “If  we  had  a complete  simulation,  we’ll  be  able  to 
see  the  temperature. ..the  flow  field  down  underneath”  surfaced 
throughout  the  instruction.  The  professor  created  new  content 
knowledge  (showing  the  lower  part  of  the  flame)  caused  by  the 
inadequacy  of  the  visual  presentation  (new  content)  to  fully  describe 
the  phenomenon  (5  incidences) 

Changed 

Changed 

Table  1.  Illustrations  of  the  three  classes  of  interaction,  transformation , adaptation,  and  negotiation  as 
exhibited  by  the  professor  in  the  8:  25  minutes  of  Metaverse  fire  whirl  presentation. 


to  obtain  the  desired  views  of  the  simulated  fire.  In  other  words,  the  learning  through  teaching  occurs  and  can 
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be  described  as  mediated  activities  that  suggest  the  fulfillment  of  the  intended  goals,  which  are  the  requisite 
pedagogical  strategies  to  support  meaningful  instruction. 


Conclusion 

This  initial  study  provides  the  frameworks  to  explore  the  development  of  pedagogical  approaches 
within  the  context  of  the  immersive,  non-restrictive  environment.  Although  the  visual  display  and  the 
immersive  environment  are  still  in  an  early  stage  of  development,  it  offers  a fertile  ground  for  research  in 
human  computer  interaction,  learning  and  instruction,  and  human  behaviors  that  may  provide  insight  in  areas  of 
cognitive  domain . 


Figure  1.  Lowering  the  body 
(tool-shown  in  bottom  left)  to 
show  the  underneath  view  of  the 
flow  field 


Figure  2.  Showing  the  fire 
whirl  data  from  the  top  view  by 
standing  (tool)  and  holding  up 
the  tracking  device  (artifact) 


Figure  3.  Lowering  the  body 
(bottom  right)  and  holding  the 
tracking  device  with  the  left  hand  to 
manipulate  the  view  of  the  display 
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Survey  Study  of  Using  Technology  in  Online  Courses 

Shijuan  Liu 
Indiana  University 

Introduction 

Compared  to  courses  delivered  in  face  to  face  setting,  courses  that  were  delivered  entirely  online  rely 
more  on  technology  (Bonk,  2001;  Firdyiwek,  1999;  Moore,  2003).  Technology,  especially  the  Internet,  provides 
a common  virtual  space  for  students  and  instructors  who  are  physically  separated;  it  is  widely  acknowledged  as 
an  essential  component  of  teaching  and  learning  environments  in  online  setting. 

As  more  and  more  technology  tools  become  available  for  online  education,  there  have  a development 
of  interest  among  educators  and  other  professionals  in  the  potential  application  of  the  tools  in  online  courses 
(Hanna,  2003;  Moore,  2003).  For  instance,  some  researchers  argued  that  a course  manage  system  (CMS),  a 
collection  of  such  software  tools  as  asynchronous  discussion  board  and  real  time  chat  tools,  had  potential  to 
transform  teaching  and  learning  (Ansorge  & Colley,  2003;  Carmen  & Haefner,  2002).  At  the  same  time, 
researchers  realized  that  technology  tools,  like  other  tools  human  beings  developed,  can  be  used  in  profound  as 
well  as  very  trivial  and  careless  ways  in  educational  practice  (Althauser,  R.  & Matuga,  J.,  1998;  Ottenhoff  & 
Lawrence,  1999).  In  addition,  instructors  have  been  acknowledged  to  play  a key  role  in  using  the  technology 
successfully  (Garrison,  Anderson,  & Archer,  2003;  Willis,  1994;  Wilson,  1998). 

What  is  the  current  state  of  online  instructors  in  using  the  technology  tools?  As  many  researchers 
pointed  out,  there  is  a pressing  need  to  study  how  the  technology  is  being  used  in  online  courses  (Bonk,  2001, 
2003;  Garrison  et  al,  2003;  University  of  Illinois,  1999).  Answers  to  this  question  not  only  help  researchers 
understand  the  current  state  of  the  online  courses,  but  also  help  tool  developers,  instructional  designers, 
instructors,  and  administrators  in  making  decisions  for  their  practices,  products,  or  services.  However,  literature 
indicated  that  our  understanding  of  the  question  was  very  limited  (Bonk,  2003;  Garrison,  et  al,  2003).  Only  a 
few  survey  studies  were  found  for  answering  the  question;  and  even  fewer  studies  were  found  to  investigate  the 
question  from  different  aspects  in  depth.  The  particular  study  was  conducted  with  an  attempt  to  make 
contributions  to  the  area. 


Purpose  of  the  Study 

The  study  attempted  to  investigate  the  use  of  technology  by  online  instructors  from  the  following 
aspects:  what  tools  were  used  by  the  instructors  in  teaching  online  courses?  What  features  of  the  tools  did  they 
use,  in  which  way,  and  for  what  purpose(s)?  How  did  they  perceive  the  importance,  necessity,  and  effectiveness 
of  the  features?  What  new  features  did  they  expect  the  tools  to  have?  Were  their  skills  of  using  the  tools  related 
to  their  perceptions  of  and  the  way  how  they  used  the  tools? 

Due  to  the  plethora  of  available  technologies  as  well  as  time  and  budget  constraints,  this  study  focused 
on  the  following  key  elements  in  the  online  courses:  asynchronous  discussion,  real  time  (synchronous)  chat, 
audio/video,  and  supporting  team  work  function.  Features  such  as  survey  functions,  use  of  e-mails  (external  and 
internal),  control  of  sharing  level  in  managing  courses  (i.e.,  giving  feedback  or  delivering  materials  to  the  whole 
class,  a team  or  an  individual)  were  also  included  in  the  study. 

Methods 

Participants 

Thirty  instructors,  who  taught  online  courses  in  the  School  of  Education  of  a large  Midwestern 
university  during  the  period  from  Spring  to  Fall  of  2003,  were  invited  to  participate  in  the  study. 

Instrument: 

An  electronic  questionnaire  was  used  that  consisted  of  ten  sections,  including  asynchronous  discussion, 
real-time  online  chat,  audio/video,  team  work,  survey  function,  control  of  sharing  level,  etc.  A five-point  scale 
was  utilized  to  ask  the  participants  to  rate  the  importance  of  the  features,  the  effectiveness  of  the  tools  they  used 
in  support  their  teaching,  and  their  levels  of  skill  in  using  some  features.  In  addition  to  the  close-ended 
questions,  there  were  eleven  open-ended  questions  included  in  the  survey.  The  open-ended  questions  asked 
participants  to  explain  why  they  thought  certain  tools  or  functions  were  necessary  or  not  necessary,  to  describe 
how  they  used  real-time  chat,  asynchronous  discussion,  video/audio,  and  team  work  in  their  courses,  and  to  list 
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the  new  features  that  they  would  like  the  tools  to  have. 


Data  Analysis 

Responses  to  the  close-ended  questions  with  a five-point  scale  were  analyzed  by  SPSS.  Percentage, 
mean,  and  standard  deviation  of  individual  items  were  calculated.  Correlations  between  or  among  some  relevant 
items  were  also  examined.  Data  collected  from  the  open-ended  questions  were  analyzed  manually.  The 
frequency  of  each  emerging  theme  was  counted,  and  the  representative  quotations  of  the  respondents  were 
selected. 

Procedure 

: The  contact  information  of  the  30  instructors  was  collected  from  the  websites  of  the  course  they 
taught.  The  questionnaire  was  sent  to  the  instructors  by  an  e-mail  attachment  in  the  October  of  2003.  In  the  e- 
rnail,  the  purposes  of  the  study  were  explained.  To  improve  the  survey  return  rate,  the  e-mail  was  personalized 
as  much  as  possible.  For  instance,  each  of  the  participants  was  addressed  by  their  name  (e.g.,  Dr.xx,  Professor 
xx),  and  the  course(s)  he  or  she  taught  was  also  mentioned  in  the  e-mail.  The  participants  returned  the  survey 
also  by  e-mail.  Many  of  the  participants  returned  the  completed  surveys  the  same  day  or  the  second  day  that  the 
survey  was  sent  out.  A thank-you  e-mail  was  sent  to  each  of  the  respondents  after  the  survey  was  returned. 

Some  respondents  were  asked  to  elaborate  some  points  further  by  a follow-up  e-mail. 

Limitations 

The  study  had  two  major  limitations.  First,  the  sample  size  was  small . This  might  be  justified  by  the 
fact  that  the  questionnaire  included  more  open-ended  questions  than  normal  survey  studies,  and  the  data  were  a 
mix  of  quantitative  and  qualitative  data.  Nevertheless,  readers  need  to  be  cautious  when  making  generalizations 
on  the  findings  of  the  study  beyond  the  sample.  Second,  even  though  the  drafts  of  the  questionnaire  were 
reviewed  by  three  experts  and  tested  by  two  associate  instructors  before  it  was  sent  out,  the  questionnaire  could 
be  further  improved.  For  instance,  some  survey  items  later  turned  out  still  not  to  be  very  specific  to  some 
respondents. 


Results  and  Interpretations 

Twenty  out  of  thirty  participants  (66.7%)  completed  the  questionnaires.  Among  them,  thirteen  were 
females.  Seven  were  males. 

Ninety  percent  of  the  respondents  were  found  to  use  the  course  management  systems  (CMS)  that  the 
university  provided,  namely,  SitesScape  Forum  (SSF)  and  Oncourse.  Only  five  percent  did  not  use  the  CMS, 
but  the  Bulletin  Board  System  (BBS). 

Asynchronous  Discussion 

Importance.  Respondents  were  asked  to  rate  the  importance  of  using  asynchronous  discussion  in  online 
courses  with  a five -point  scale  (l=lowest  importance,  5=highest  importance).  Whereas  5%  of  the  respondents 
rated  the  importance  as  the  lowest,  70%  of  them  rated  it  as  the  highest.  On  average,  the  respondents  perceived 
using  asynchronous  discussion  as  being  important  or  very  important  in  online  courses  (M=  4.4,  SD=1.4). 

How  it  was  used.  Respondents  who  used  asynchronous  discussion  in  online  courses  described  the  way 
they  used  it.  Their  responses  were  listed  according  to  the  frequency  of  the  themes  from  high  to  low:  (1)  students 
shared  their  reading  reaction,  experience  and  got  support  from  peers  and  mentors/instructors;  (2)  assigned  roles 
(facilitators/leaders  and  wrappers/summarizers,  instigators,  devil’s  advocate)  to  students,  word  count  for  the 
discussion;  (3)  the  instructor  raised  some  discussion  questions;  students  responded  to  the  questions  and 
responded  to  each  other;  (4)  peer-review  posted  projects  and  assignments;  (5)  students  posted  their  reading 
reaction  first,  then  were  responsible  for  responding  to  a certain  number  of  their  classmates’  postings;  and  (6) 
students  discussed  in  teams  of  4 or  5 at  the  end  of  the  discussion  period;  each  team  posted  a summary  of  their 
discussion  to  share  with  the  whole  class.  The  duty  of  facilitator  rotated  among  the  team  members.  Students 
completed  peer  evaluations  on  their  peer  performance.  The  instructor  monitored  the  discussion  and  added 
occasional  comments,  especially  if  s/he  thought  they  were  getting  off  track. 

Effectiveness.  Participants  were  asked  to  rate  the  effectiveness  of  the  tool(s)  that  they  used  in 
supporting  online  asynchronous  discussion  with  a five-point  scale  (l=least  effective,  5=highest  effective). 
Results  showed  that  none  of  the  respondents  rated  the  effectiveness  as  the  lowest,  while  44.4%  of  them  rated  it 
as  the  highest.  On  average,  the  respondents  believed  that  the  tools  they  used  were  effective  (M=4.11,  SD=.96). 
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New  functions.  Participants  were  surveyed  on  what  new  functions  that  they  would  like  the  tool(s)  to 
have  and  which  suggestions  they  might  have  for  improving  the  tools  they  used.  Their  responses  could  be 
divided  into  two  categories:  (1)  pedagogy  related:  ability  to  hide  posts  until  a student  posts  themselves  (to 
ensure  the  first-level  postings  are  original,  not  borrowing  from  others);  ability  to  hide/change  name  for 
anonymity  in  discussions  periodically;  ability  to  easily  create  discussion  group  space;  track  of  read  and  unread 
messages  well;  automatically  formulate  folders  instead  of  thread;  and  ability  to  support  graphic  and  video;  (2) 
usability  related:  ability  to  recall,  delete,  edit  one’s  own  messages;  better  threading  of  the  discussion;  a more 
user-friendly  interface  design;  easier  printability  and  navigation,  and  ability  to  simultaneously  view  the  posting 
to  which  one  was  replying. 

Real-time  (synchronous)  Chat 

Real-time  chat  tools  that  the  respondents  used  included:  Tapped-in,  AOL  instant  messenger,  MSN 
messenger,  SchMooze,  and  the  chat  function  of  the  course  management  systems  (i.e.,  SSF  or  Oncourse). 

Necessity.  When  asked  about  whether  it  was  necessary  to  use  real-time  chat  in  online  courses,  30% 
said  “no”;  25%  said  “yes”;  25%  of  the  respondents  said  it  could  be  beneficial,  but  had  some  problems;  15%  said 
it  would  depend  on  the  characteristics  of  students,  instructional  goals,  and  technology  consideration(s);  One 
respondent  indicated  that  she  did  not  know  because  she  had  not  used  this  before.  Reasons  that  the  respondents 
gave  for  why  it  was  necessary  to  use  could  be  put  into  two  categories:  (1)  students  felt  more  comfortable  since 
real-time  chat  was  more  informal;  it  brought  in  some  authenticity,  and  helped  build  a sense  of  community;  and 
(2)  it  was  efficient  to  use  real-time  chat  to  communicate  and  give  immediate  feedback.  Problems  and  concerns 
that  the  respondents  who  held  a negative  or  neutral  position  in  using  the  real-time  chat  were  listed  as  follows 
based  on  the  order  of  the  frequency  from  high  to  low:  (1)  it  was  difficult  to  arrange  for  both  instructor  and 
students  because  students  were  from  different  time  zones  and  had  different  schedules;  (2)  asynchronous 
discussion  was  more  important  because  it  forced  students  to  reflect  more  and  provide  them  more  time 
flexibility;  (3)  it  would  take  away  one  of  the  advantages  of  taking  online  courses;  and  (4)  it  would  be  an 
unreasonable  burden  for  students. 

How  it  was  used.  Respondents  who  reported  that  they  used  real-time  chat  in  their  online  courses 
described  the  way  they  used  it.  Their  responses  were  listed  according  to  the  order  of  frequency  from  high  to 
low:  (1)  had  students  share  their  ideas  for  their  own  or  group  project  and  get  feedback  from  the  instructor  and 
peers;  (2)  had  students  visit  virtual  environment  to  explore  possibilities  of  use  in  their  [student  Jteaching;  (3) 
used  Instant  Messenger  for  office  hours:  Students  can  “pop  in”  to  ask  questions;  and  see  when  students  are 
online  and  remind  them  of  things;  (4)  used  for  interaction  with  a guest  speaker;  (5)  introduced  everyone  to  each 
other  at  the  very  beginning  of  semester,  and  to  answer  any  concerns  that  have  come  up  in  terms  of  using  the 
tool;  (6)  used  for  unit  wrap-ups,  dis  cussed  the  main  topics  of  that  week,  and  clarified  assignments;  (7)  used  for 
readjust  course  schedule;  and  (8)  used  for  personal  communication. 

Rating  of  skills.  Participants  were  asked  to  rate  their  skills  of  using  real-time  chat  with  a five -point 
scale  (l=lowest,  5=highest).  Two  out  of  the  twenty  respondents  did  not  answer  the  question.  Those  two 
respondents  had  previously  indicated  that  they  had  not  used  the  real-time  chat  and  would  not  consider  using  it  in 
the  future.  Among  the  eighteen  respondents,  33.3%  of  them  rated  their  skills  as  lowest,  while  27.8%  of  them 
rated  their  skills  as  highest. 

Audio/Video 

Necessity.  Participants  were  asked  whether  it  was  necessary  to  use  audio/video  in  online  courses  and 
why  they  thought  that  way.  Only  one  out  of  the  twenty  respondents  (5%)  said  “yes”.  The  reasons  he  stated  were 
as  follows:  “it  helps  enhance  the  authenticity  of  a learning  environment  and  create  a psychological  proximity  in 
the  geographically  distributed  learning  community.  In  addition,  both  audio  and  video  can  enhance  students 
understanding  for  learning  concepts  and  principles,  which  is  otherwise  explained  through  heavy  text.  As  the 
majority  of  the  students  had  high-speed  connection,  downloading  was  not  an  issue.”  Another  respondent  (5%) 
said  he  did  not  know  because  he  had  not  used  audio/video  in  online  courses  before.  20%  of  the  respondents 
chose  “no”.  The  reasons  given  included:  “Current  resources  (without  using  audio/video)  are  sufficient  and 
effective  ‘in  supporting  students’  efforts  to  meet  course  objectives”;  “Audio  did  not  add  much  to  course 
management.  Asynchronous  conferencing  has,  as  literature  also  tells  us,  advantages  and  that  is  good  enough 
(reflective  & critical  thinking,  etc.)  for  my  purposes  of  attaining  high-level  learning”;  “Many  people  like 
anonymity”.  Thirty  percent  of  the  respondents  believed  it  would  depend  on  the  course  content,  students  (needs), 
instructor,  circumstance,  etc.  As  one  of  the  respondent  said,  “In  my  classes,  I don't  really  think  it  is  necessary. 
But  for  other  online  courses  that  gear  toward  multimedia  design  and  other  visual/auditory -oriented  subjects, 
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audio/video  supplement  might  be  helpful.”  Forty  percent  of  the  respondents  believed  that  it  was  not  absolutely 
necessary,  but  it  could  be  useful,  fun,  and  could  increase  class  interactivity,  etc.  In  the  words  of  one  respondent: 
‘‘It’s  not  necessary,  but  it  can  be  a useful  tool....  some  things  are  easier  to  show  in  a video  clip,  for  instance, 
than  to  describe  in  words.” 

How  it  was  used.  If  they  had  used  audio/video  in  their  courses,  participants  were  asked  to  describe  the 
way  they  used  it  and  the  students’  responses  to  it.  Five  respondents  answered  this  question.  One  respondent 
reported  that  she  used  a stream  video  which  was  composed  of  interviews  of  online  teachers  and  students  about 
their  experiences;  According  to  her,  students  really  liked  it,  but  she  had  to  send  students  a CD  ROM  via  “snail 
mail”  because  about  25%  of  them  had  technical  problems.  Another  respondent  said  that  he  used  audio  and  video 
clips  as  examples  since  the  course  was  about  multimedia  production,  and  students  were  required  to  make  audio 
and  video  clips  on  their  own.  However,  he  was  not  sure  whether  students  liked  the  clips  or  not.  Still  another 
respondent  reported  that  he  created  an  audio  clip  of  lecture  because  he  could  not  find  a web  resource  with 
relevant  information.  He  had  assumed  that  audio  would  be  more  interesting  than  reading  a long  document; 
however  was  surprised  to  receive  complaints  from  some  students  because  the  clip  did  not  have  visual  elements. 

Consider  using  it  in  the  future.  Participants  who  had  not  used  audio/video  in  their  courses  were  asked 
whether  they  would  consider  using  it  in  the  future.  Almost  42  percent  (41.6%)  of  the  respondents  said  “yes” 
based  with  the  reason  that  it  could  enhance  learning  and  enrich  the  online  classroom  to  meet  the  learning  styles 
of  different  students.  One  of  them  said  that  she  would  learn  how  to  use  it,  if  it  was  indeed  proven  useful.  16.7% 
said  they  would  not  consider  using  it  in  the  future.  The  rest  of  the  respondents  reported  that  whether  they  would 
consider  using  it  depended  on  students’  needs.  As  one  of  them  said,  “It  would  have  to  really  be  worthwhile  for 
me  to  use  it,  rather  than  a high-tech  option.” 

Teamwork 

Necessity.  Sixty-five  percent  of  the  respondents  believed  that  it  was  necessary  to  use  team  work  in 
online  courses.  The  reasons  they  listed  included:  (1)  it  forced/encouraged  students  to  learn  from  each  other,  and 
take  advantages  of  the  learning  community;  (2)  it  helped  students  to  get  to  know  one  another  and  increase  the 
feeling  of  community;  (3)  it  addressed  affective  needs;  (4)  it  helped  students  to  process  new  information  more 
effectively;  (5)  it  encouraged  students  to  keep  up  with  the  course;  and  (6)  it  helped  students  to  try  out  their  ideas 
in  a low-stakes  setting.  Fifteen  percent  of  the  respondents  said  it  was  absolutely  not  necessary  based  on  the 
following  reasons:  (1)  students  often  had  different  schedules  and  were  working  on  different  parts  of  the  course; 

(2)  teamwork  was  time  -consuming.  Since  students  were  from  different  places  with  busy  schedule,  there  were 
usually  challenges  to  overcome  before  the  team  work  even  took  place;  and  (3)  team  work  was  only  one 
potential  pedagogical  option  in  any  course  (online  or  face-to-face).  While  it  had  benefits,  it  also  created  some 
problems.  Students  can  still  effectively  interact  without  having  to  be  on  a team  project.  Twenty  percent  of  the 
respondents  held  a neutral  position,  believed  that  it  was  not  necessary  but  a plus,  or  indicated  that  it  would 
depend  on  a lot  of  things  such  as  of  the  topic  and  the  instructional  style  of  an  instructor. 

How  it  was  used.  Fourteen  respondents  described  how  they  used  the  team  work  in  online  courses.  The 
ways  included:  (1)  using  teamwork  for  discussion,  peer  review  and  giving  each  other  feedback;  (2)  giving 
student  certain  roles  in  doing  the  teamwork;  and  (3)  requiring  students  to  provide  a team  product  and  did  team 
presentation. 

Effectiveness  of  supporting  teamwork.  On  average,  the  respondents  perceived  the  effectiveness  of  the 
tools  they  used  in  supporting  teamwork  as  moderately  high  (M=3.4,  SD=1.0). 

New  functions  to  be  added.  Suggestions  that  respondents  given  for  making  the  tools  to  support  team 
work  more  effectively  included:  add  a white  board;  offer  synchronous  video  conference;  and  to  improve  the 
functions  of  using  team  space. 

Other  Features 

The  study  also  surveyed  other  features  such  as  survey  functions,  e-mails,  and  control  of  sharing  level. 
Results  indicated  that  respondents  did  not  perceive  the  survey  function  as  important  (M=2.3,  SD=1.26).  They 
used  the  survey  function  mainly  in  the  following  ways:  (1)  collecting  formative  and/or  summative  evaluation 
from  students  about  the  course  design,  teaching,  and  learning  satisfaction;  (2)  setting  synchronous  meeting  time; 

(3)  finding  out  students’  level  of  experience  with  technology  in  the  first  week  of  class;  and  (4)  asking  students 
at  the  beginning  of  the  course  what  they  hope  to  get  from  this  course. 

Results  also  indicated  that  the  feature  of  controlling  sharing  level  in  online  courses  was  perceived  as 
important  (Mean=3.9,  SD=1.20).  Regarding  the  internal  e-mail  function  that  the  course  management  system 
provided,  most  respondents  perceived  its  importance  as  moderately  low  (M=2.83,  SD=1.20).  In  contrast,  the 
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importance  of  using  regular  e-mails  in  teaching  online  courses  was  perceived  as  being  high  (M=4.05,  SD=.78). 
Due  to  the  space  limit,  data  on  other  features  that  were  also  collected  with  the  questionnaire  were  not  reported 
in  the  paper,  but  are  available  upon  request. 

Discussion  and  Implications 

As  noted,  almost  all  the  respondents  in  the  study  were  found  to  use  course  management  systems  in 
their  online  courses.  This  finding  is  consistent  with  what  other  studies  found  (e.g.,  Bonk,  2001;  Teles,  2003).  It 
also  added  empirical  data  to  the  literature  that  CMS  has  become  mainstream  practice  in  online  courses  (e.g., 
McLoughlin  & Luca,  2000).  The  popular  use  of  the  CMS  has  implications  for  tool  developers,  university 
administrators,  instructional  designers,  and  technical  support  staff.  For  instance,  they  need  to  continue  to 
develop  and  improve  the  CMS,  and  provide  support  for  using  it. 

Asynchronous  discussion,  also  known  as  asynchronous  communication,  threads  discussion,  and 
delayed  computer  conferencing  in  the  literature,  has  been  used  for  over  a decade  (Garrison,  et  al,  2003).  Its 
advocateors  argued  that  it  supported  greater  independence  and  flexibility  from  temporal  and  geographical 
barriers  and  provided  more  reflective  participation  (Feenberg,  1989).  The  respondents  were  found  to  have  a 
high  consensus  toward  the  importance  of  using  it  in  online  courses.  Eighty -five  percent  of  them  perceived  it  as 
the  most  important  or  very  important.  The  finding  was  consistent  with  the  relevant  literature  and  studies  (Bonk, 
2001;  Bonk  & King,  1998a,  1998b;  Garrison  et  al,  2003;  Teles,  2003).  Literature  showed  that  asynchronous 
discussion  could  be  used  in  following  ways:  general  discussions,  exchanging  ideas,  working  on  specific  topic 
areas,  and  peer  commenting  (Siegel  & Kirkley,  1998;  Kang,  1998).  The  way  the  respondents  used  it  was 
supported  by  the  literature.  Some  respondents  assigned  students  different  roles  in  doing  weekly  discussion,  and 
some  asked  students  to  facilitate  the  discussion  in  teams.  This  helped  students  master  the  content  and  develop 
their  collaboration,  reflection  and  critical  thinking  skills  (Duffy,  Dueber  & Hawley,  1998;  Zhu,  1998).  This 
study  indicated  that  most  instructors  become  familiar  with  using  asynchronous  discussion  and  are  relatively 
skillful  with  it.  Duffy  et  al.  (1998)  criticized  that  “many  designers  of  conferencing  systems  have  had  a simplistic 
view  of  discussion  as  simply  talking”  (p.74),  and  argued  for  a more  effective  and  pedagogical-based 
conferencing  system  to  support  online  asynchronous  discussion.  It  is  worth  noting  that  at  the  present  time  some 
conferencing  systems  have  developed  a number  of  pedagogical  features  suggested  by  Duffy  et  al  (1999).  For 
instance,  SSF  enables  instructors  to  track  the  history  of  each  post  and  see  who  has  read  it,  thereby  providing  the 
instructors  with  yet  another  index  of  student’s  participation.  The  respondents  in  the  study,  on  average,  believed 
that  the  current  asynchronous  discussion  tools  were  effective  in  supporting  their  teaching.  However,  based  on 
the  respondents’  suggestions  it  still  seemed  to  be  a need  for  the  designers  and  developers  to  further  improve  the 
tools  in  both  pedagogical  and  usability  aspects. 

According  to  the  literature,  real-time  chat,  also  known  as  synchronous  discussion  or  synchronous 
communication,  was  criticalfor  online  courses.  Its  advocators  argued  that  unlike  the  delayed  exchange 
(asynchronous  communication),  real-time  chat  provided  “teachers  and  students  with  a forum  for  an  immediate 
and  dynamic  interchange  of  ideas”,  and  “can  be  an  exciting  asset  to  collaborative  learning  environments” 
(Cooney,  1998.  p.263).  It  can  be  used  to  foster  group  cohesion  and  decision  making,  brainstorming,  and  build 
high  levels  of  socialization  (Kang,  1998;  Roberts,  as  cited  in  Garrison  et  al,  2003).  The  respondents  in  the  study 
who  believed  using  real-time  chat  was  necessary  shared  the  same  view,  especially  on  its  efficiency  of  giving 
immediate  feedback  and  helping  build  a sense  of  community.  How  they  used  it  was  also  consistent  with  what 
the  literature  suggested.  Such  strategies  as  using  it  for  office  hours  and  interaction  with  a guest  speaker 
demonstrated  the  advantages  of  the  real-time  chat  and  the  instructors’  skills  of  using  it  in  a very  thoughtful  and 
effective  manner.  However,  two -third  of  the  respondents  held  a neutral  or  negative  position  on  using  real-time 
chat.  The  problems  and  concerns  that  they  listed  (e.g.,  hard  to  arrange  because  of  different  time  zones  and 
schedule)  were  reasonable.  Administrators,  instructional  designers,  and  support  staff  members  seem  to  be  able 
do  at  least  two  things  in  order  to  help  the  instructors  who  held  similar  position  to  take  advantages  of  using  real- 
time chat.  One  is  to  help  them  realize  what  the  advantages  are;  another  is  to  provide  them  with  strategies  on 
how  to  tackle  the  problems  and  help  solve  concerns  that  they  have  in  using  it.  In  addition,  the  finding  of  the 
study  indicates  that  low  skill  and  lack  of  experience  is  another  barrier  for  the  instructors  who  did  not  utilize  the 
technology  well.  For  instance,  the  respondents  who  rated  their  skills  as  the  lowest  were  also  those  who  had  not 
used  it.  This  further  proved  the  need  to  expose  the  instructors  to  how  to  use  the  real  time  chat  in  both  the 
technical  and  instructional  sense. 

As  reported  earlier,  one  respondent  who  supported  using  of  audio/video  listed  the  following  advantage 
of  using  it.  e.g.,  helping  enhance  the  authenticity  of  the  learning  environment,  create  a psychological  proximity, 
and  enhance  students’  understanding  of  learning  concepts  and  principles.  However,  the  finding  of  the  study 
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indicated  that  the  advantages  seemed  not  to  have  been  widely  acknowledged.  Among  the  20  respondents  only 
one  chose  “yes”  when  asked  about  the  necessity  of  using  the  technology.  In  addition,  the  technology  did  not 
seem  to  be  utilized  to  its  best  capacity.  Kirschner  (1991)  found  that  using  audio  clips  as  one  way  to  give 
feedback  to  student  assignments  was  perceived  as  being  of  higher  quality  than  text  -based  feedback.  Provision  of 
feedback  with  audio  also  took  less  instructor  time.  Among  the  only  five  respondents  who  described  the  way 
how  they  used  in  their  courses,  however,  none  of  them  indicated  that  they  used  the  technology  to  give  feedback 
or  communicate  with  students.  The  finding  implied  that,  like  in  the  case  of  the  real-time  chat,  administrators, 
instructional  designers,  and  other  support  staff  members  need  to  provide  online  instructors  with  more 
opportunities  to  realize  its  advantages  and  to  help  them  improve  instructional  strategies  in  using  it.  It  is  worth 
pointing  out  that  compared  to  their  responses  to  real-time  chat,  respondents  who  had  not  used  the  technology 
seemed  to  be  more  willing  to  consider  using  audio  and  video  technology  in  the  future.  When  surveyed  for  what 
new  features  needed  to  better  support  asynchronous  discussion,  one  respondent  listed  the  ability  to  support 
video.  This  is  consistent  with  what  some  researchers  found.  Garrison  et  al.  (2003)  argued  that  “as  tools  become 
more  efficient  and  less  costly  and  bandwidth  becomes  cheaper  and  more  widely  available”,  “most  new 
computers  come  equipped  with  microphones  and  recording  software,  ad  video  recording  systems  can  be 
purchased  and  install  on  newer  computers  for  under  $200”,  there  will  be  “considerable  value”  in  adding  such 
multimedia  technology  to  the  CMS  (p.l  19). 

The  advantages  of  using  teamwork,  also  called  as  group  work  or  collaboration,  has  been  discussed 
widely  in  the  literature  (Johnson  & Johnson,  1996).  The  majority  of  the  respondents  perceived  using  teamwork 
as  necessary  in  online  courses.  The  reasons  they  gave  reflected  that  their  positions  were  grounded  on  thoughtful 
pedagogical  considerations.  The  various  ways  of  using  teamwork  that  the  respondents  described  indicated  that 
the  respondents  were  experienced  on  this  regard.  It  is  notable  that  compared  to  the  rating  of  the  effectiveness  of 
the  tools  in  supporting  asynchronous  discussion,  the  effectiveness  of  the  tools  to  teamwork  was  rated  lower.  It  is 
also  interesting  to  notice  that  one  respondent  suggested  adding  synchronous  video  conferencing  to  make  the 
tools  better  support.  One  suggestion  given  to  better  support  asynchronous  discussion  addressed  the  video 
element  as  well. 

When  compared  the  respondents’  perceptions,  it  was  found  that  their  perceptions  on  the  four  elements 
varied.  If  the  perceived  importance  (or  necessity)  of  using  the  elements  are  regarding  as  a continuum  with  the 
most  important  (necessary)  in  one  end,  and  the  least  important  and  not  necessary  on  the  other  end,  then 
respondents’  perception  on  using  asynchronous  discussion  and  teamwork  will  be  above  the  middle  point 
(neutral  position)  and  more  toward  the  end  of  most  important  (or  necessary)  side.  In  contrast,  using  real-time 
chat  and  audio/video  will  be  lower  than  the  neutral  position,  and  toward  the  end  of  least  important  (or 
necessary)  side.  In  addition,  the  study  also  found  that  the  respondents’  position  on  the  importance  or  necessity 
of  using  certain  tools  was  related  to  the  extent  that  they  perceive  the  benefits  (or  advantages)  and  challenges  (or 
problems)  of  using  the  tools.  Specially,  the  more  benefits  they  emphasized,  the  more  likely  they  perceived  the 
tools  as  important  or  necessary  to  use.  In  other  words,  their  position  will  tend  to  be  positive.  If  they  put  more 
emphasis  on  the  challenges,  their  position  will  be  more  likely  to  be  negative.  In  addition,  the  study  indicated 
that  there  was  a positive  correlation  between  the  instructors’  skills  of  using  certain  tools  and  how  likely  they 
have  used  or  will  use  them.  There  was  also  a positive  correlation  found  between  the  perceived  importance  and 
how  often  the  tools  were  used.  As  discussed  earlier,  this  has  implications  for  the  online  administrators, 
instructional  designer,  and  other  supporting  staff  members.  Workshops  on  helping  the  instructors  better 
understand  the  advantages  and  disadvantages  of  using  the  tools,  and  how  to  use  them  technologically  and 
pedagogically  are  needed. 


Summary  and  Conclusions 

The  study  attempted  to  investigate  the  current  state  of  how  instructors  use  technology  in  online  courses 
with  a focus  on  use  of  the  four  key  elements.  Namely,  asynchronous  discussion,  real-time  chat,  audio/video,  and 
team  work.  Major  findings  include  that  asynchronous  discussion  and  team  work  were  perceived  as  being  very 
important  or  necessary  to  be  used  in  online  courses;  while  audio/video  and  real  time  chat  were  perceived  as  less 
important  or  less  necessary.  The  way  the  instructors  used  the  technologies  in  their  courses  were  reported  and 
was  connected  the  relevant  literature.  The  study  also  identified  some  relations  between  the  instructors’ 
perceptions  and  their  practices,  between  their  skills  of  using  certain  tools  and  how  often,  and  how  likely  they 
used  them.  The  findings  of  the  study  will  help  add  empirical  data  to  the  relevant  research,  and  help  online 
administrators,  instructional  designers,  instructional  and  technical  support  staff,  and  tool  developers  with 
offering  better  tools,  appropriate  workshops,  and  corresponding  support. 
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Abstract 

The  complexities  of  digital  age  pose  challenge  to  existing  instruction  technology  theory  as  it  applies  to 
a distance  learning  environment.  Through  the  lens  of  Activity  Theory,  this  study  takes  a broad  picture  of  an 
online  course  and  examines  the  socio-cultural  factors  affecting  the  success  of  a distance  course  as  well  as  their 
complex  relationships.  Interventions  at  the  group  and  organizational  levels  will  be  explored  to  balancing  local 
tensions  emerged  in  the  implementation  of  a distance  course 

Introduction 

Computer-Mediated  Communication  offers  great  flexibility  and  accessibility  in  an  online  learning 
environment.  Enormous  elearning  demands  place  distance  learning  from  a previously  marginal  field  to  the 
central  field  of  education(Moore  & Anderson,  2003) . However,  distance  educators  and  administrators 
increasingly  realize  the  internet-supported  technology  offers  much  more  than  online  classroom  (Hara  & Kling, 
2000). In  the  most  recent  Handbook  of  Distance  Education  (DE),  it  is  stated  "....distance  education  holds  the 
promise  of  better  teaching,  better  quality  of  learning,  and  far  better  returns  to  public  and  private  institutions  for 
money  invested  education  and  training.  None  of  this  can  happen  without  careful  and  deliberate  planning, 
without  a vision  and  clear  policy."  (Moore  & Anderson,  2003).  However,  to  plan  distance  education  is  not  easy 
without  a thorough  understanding  of  the  complex  issues  involved  in  DE.  Previous  research  on  online  learning 
has  placed  much  effort  on  comparative  studies  which  usually  produce  “no  significant  differences”  results  or 
mixed  results  that  have  no  practical  values  for  improving  the  effectiveness  of  distance  learning  (The  institute  for 
higher  education  policy,  1999).  A survey  of  current  DE  research  literature  noted  several  limitations:  limited 
theoretical  framework,  failure  to  describe  the  complexities  of  the  dynamics  of  online  learning  and  lack  of 
qualitative  analysis  of  social-cultural  factors  of  distance  learning  (Berge  & Mrozowski,  2001;  Holmberg,  1987; 
Saba,  2000). 

There  is  a shift  in  educational  research  from  focusing  on  how  individuals  function  in  different  group  or 
activity  settings,  to  targeting  the  group  itself  as  the  unit  of  analysis  (Dillenbourg,  Baker,  Blaye,  & O’Malley, 
1996  as  cited  in  Bonk  & Wisher,  2000).  Schwen  (2001)  argues  the  prevalent  instructional  technology  theory  is 
often  “implicitly  or  explicitly  linked  to  a micro  instructional  theory”  which  are  unable  to  explain  the 
complexities  of  the  digital  age.  This  research  will  use  Activity  Theory  (AT),  a meso  level  theory,  as  a 
theoretical  lens  to  illuminate  how  social,  cultural  and  organizational  factors  implicitly  or  explicitly  embedded  in 
the  structure  of  a distance  course  influence  the  successful  online  learning  experience.  AT  examines  how  a group 
of  people  collaboratively  work  toward  a common  object  within  the  context  of  a community  that  is  mediated  by 
rule,  tools  and  divisions  of  labor  (Engestrom  2001).  From  AT  perspective,  learning  activities  cannot  be  fully 
understood  without  understanding  the  social  or  institutional  contexts  for  learning.  AT  suggests  studying  human 
practice  in  a social  and  historical  context  and  emphasizes  the  interaction  of  human,  social,  technological  and 
organizational  behavior  of  human  practice. 

In  this  study,  we  examined  systematically  the  socio-cultural  factors  affecting  the  success  of  a distance 
course  and  their  complex  relationships  through  the  lens  of  Activity  Theory.  Using  Activity  Theory,  this  study 
also  looked  at  the  systemic  tensions  of  distance  learning  unit  and  explored  opportunities  for  refining  existing 
practices  to  support  a distance-learning  course. 


Methods 

The  study  was  an  instrumental  case  study  in  that  the  overall  goal  will  be  to  provide  better 
understanding  or  theorizing  of  a complex  system  (Stake,  1994).  The  unit  of  analysis  identified  in  this  study  was 
a well -reviewed  distance  courses  in  an  accredited  online  MBA  program.  Two  cases  were  examined  in  this 
study.  Two  courses  were  both  core  MBA  courses.  One  course  was  Accounting  Management  and  Decision 
Making.  Another  course  was  Strategic  Marketing  Management. 
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Semi -structured  interviews  were  used  as  primary  methods  for  data  collection.  The  interview  subjects 
involve  with  various  stakeholders  of  the  course,  including  the  instructor,  all  students,  technical  support  staff, 
and  academic  advisor.  More  than  30  participants  were  interviewed  in  this  study  for  two  to  three  times  each.  15 
students  from  Accounting  Management  and  Decision  Making  and  13  students  Strategic  Marketing  Management 
participated  in  this  study.  Four  instmctors,  the  chairperson,  the  technical  support  staff  and  other  administrative 
support  were  also  interviewed.  The  interview  questions  examined  the  participant  goals  in  the  course,  social 
rules,  program  policies,  allocations  of  responsibilities,  and  organizational  operational  process,  concerned  with 
the  delivery  of  this  course. 

Meanwhile  the  researcher  examined  all  online  documents  or  offline  that  the  researcher  had  access  to 
for  the  study,  including  course  syllabus,  archived  online  asynchronous  or  synchronous  discussion  s cripts,  and 
email  communications.  In  addition,  the  researcher  observed  the  progress  of  the  course  through  online  course 
management  system  to  grasp  some  emerging  issues  that  occurred  in  the  process  of  delivering  the  course.  The 
documents  and  observations  were  used  to  provide  more  specific  details  of  the  historical  or  contextual 
information  of  the  case  and  the  people  under  study  as  well  as  to  triangulate  emergent  data  in  the  interviews  and 
to  "corroborate  and  augment  evidence  from  other  sources"  (Yin,  2003). 

Findings 

Two  cases  that  were  examined  in  this  study  seemed  to  have  different  nature  and  course  design. 
Accounting  Management  and  Decision  Making  involved  more  numerical  analysis,  and  Strategic  Marketing 
Management  course  was  more  discussion-oriented  with  case-based  learning  as  major  instruction  method. 
However,  the  socio -cultural  issues  emerging  from  the  study  were  strikingly  similar.  The  findings  below  were 
presented  under  several  categories,  including  learner  factors,  instructor  factors  and  course  contextual  factors. 

Learner 

The  students  were  unanimous  in  confirming  that  the  flexibility  of  online  learning  was  the  most  selling 
point  for  them  to  participate  in  online  learning.  While  in  their  mid  thirties  and  forties,  their  life  was  facing  a lot 
of  responsibilities,  and  the  flexibility  seemed  to  be  the  most  attractive  feature  of  online  learning  compared  with 
other  educational  options.  Normally  they  had  three  choices  for  further  education  to  advance  their  career, 
traditional  full-time  MBA,  part-time  evening  MBA  and  video-based  distance  education.  However,  the  busy 
work  schedule  did  not  allow  them  to  take  a leave  from  work  to  enter  a full-time  MBA  program.  To  enroll  in  a 
part-time  program  was  not  flexible  and  efficient  after  a whole  long  day  work.  Video-delayed  program  lacked 
interactive  discussions  with  instructors  and  fellow  students.  Online  learning  seemed  to  be  most  viable  learning 
options  for  the  participants  in  this  study. 

Activity  theory  suggests  that  the  participants  are  often  concurrently  involved  in  several  activity  systems. 
Each  activity  system  has  its  own  rules,  tools  and  division  of  labors.  There  will  be  conflicts  and  negotiations 
among  different  roles  and  rules  of  the  participants  (Engestrom  2001).  MBA  students  were  extremely  busy 
working  professionals.  They  were  facing  the  expectations  and  responsibilities  of  multiples  communities:  work 
environment,  online  learning  and  family.  Analysis  of  interviews  indicated  the  tensions  among  different  roles 
and  responsibilities  of  online  learner  in  their  work,  online  learning  and  family  activity  systems  appeared  to  be 
one  of  the  core  tensions  in  online  MBA  courses.  At  any  point  in  their  life,  their  behavior  was  a constant 
balancing  and  negotiation  act.  Two  students’  quotes  below  described  the  competing  responsibilities  of  their  life 
as  an  online  learner. 

By  working  under  the  "put  out  the  biggest  fire  first"  mentality.  It's  the  only  way  to  get 
through,  constant  re-prioritizing  all  the  demands. 

Probably  it  is  just.  I don ’t  know  if  it  is  this  course,  if  it  is  what  else  is  going  on.  In  my  life  right 
now,  you  know,  school  work,  other  classes,  spring  time  coming  up  or  what,  the  biggest  thing 
for  me  in  this  course  is  just  being  time  management.  It  is  hard  enough  sometimes  to  get  your 
own,  finding  time  to  do  assignment.  Having  to  find  time  that  both  work  for  you  and  four  other 
people  to  get  the  assignment  put  together.  That  has  been  difficulty.  There  are  so  many 
assignments.  Like  I said,  two  or  three  postings  a week  that  have  been  turned  in.  The  greatest 
challenge  is  just  time  management. 

The  participants  consistently  noted  their  motivation  to  obtain  MBA  degrees  for  a complete  package  of 
their  education  and  for  a better  job  prospect.  They  unanimously  seek  online  programs  with  good  reputation  to 
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ensure  that  their  credentials  were  guaranteed.  Several  metaphors  were  used  frequently  by  the  participants  to 
describe  the  importance  of  credentials,  such  as  “check  box  approach”,  “company  ladder”,  “punch  the  ticket”. 
Thus,  ranking  became  an  important  criterion  for  them  to  choose  online  programs  because  for  them,  good 
ranking  means  good  quality  of  education  and  good  job  prospect.  One  of  the  participants  in  this  study  elaborated 
on  the  importance  of  reputation  of  an  online  MBA  program. 

7 wanted  to  do  MBA  to  have  a better  job  prospect.  1 tried  for  admission  for  full-time  MBA  in 
top  3 MBA  programs  in  US  but  didn 't  get  admission.  Then  I started  looking  for  distant 
programs  that  are  reasonably  priced  for  me  and  have  brand  name.  Online  program  has  the 
advantage  of  moving  from  one  geographic  area  to  another  without  abandoning  the  MBA 
program.  From  research  in  Business  Week's  website,  I found  Kelley  Direct  satisfies  my 
criteria  best  and  fulfills  my  needs.  Hence  I joined  Kelley  Direct  MBA  program. 


Instructor 

Rather  than  viewing  online  teaching  as  a transformation  from  traditional  classroom,  the  instructors  of 
two  cases  still  viewed  online  learning  activity  system  as  historically  new  development  of  traditional  activity 
system.  However,  the  introduction  of  new  online  tools  caused  disturbances  in  the  instructors’  beliefs  about  their 
roles  in  a traditional  setting  and  thus  made  them  to  make  adjustable  actions.  By  trying  to  adjust  to  a new 
environment,  instructor’s  philosophies  and  beliefs  about  online  learning  were  reflected  from  several  influences: 
cultures  and  communities  of  profession,  historical  influence  of  traditional  teaching,  perceived  attributes  of 
online  tools  and  new  rules  and  according  adapting  strategies. 

The  instructors  expected  to  give  online  MBA  students  equivalent  education  than  traditional  students. 
The  new  activity  system  of  online  courses  retained  the  same  objective  and  teaching  methodologies  they  used  in 
traditional  classroom.  One  advantage  of  online  learning  perceived  by  the  instructors  was  that  multiple  identities 
of  online  students  gave  them  a learning  advantage  than  traditional  student.  For  example,  one  professor  from  the 
Marketing  Strategic  Management  course  commented, 

So  our  final  assignment  takes  them  right  specific  to  their  company  and  hopefully  the  outputs 
in  their  specific  job  and  we  try  to  set  them  up  by  saying  the  assignment  on  the  final  project  is 
to  make  at  least  one  implemental  recommendation  for  an  improvement  and  we  encourage 
them  to  show  it  to  other  people  in  the  company  because  they  have  to  interview  them  in  that 
context  and  that  sort  of  bolsters  their  ability  to  take  the  stuff  which  is  hopefully  become  less 
abstract  in  the  course  and  apply  it  very  close  to  home  and  hopefully  to  obtain  professional 
recognition  in  -house  and  that  has  happened  sometimes. 

The  loss  of  social-contextual  cues  in  the  online  activity  system  caused  disturbances  on  traditional 
teaching  process.  Without  the  dynamics  in  a classroom,  the  instructors  felt  it  is  difficult  to  understand  and 
engage  students  in  the  same  way  as  in  traditional  classroom.  One  of  the  instructors  commented  on  how  he  tried 
to  use  the  explicit  and  exaggerated  language  to  create  a warm  environment  but  perceived  it  as  an  imperfect  way 
to  do  this  in  an  online  course. 

Within  the  direct  classroom  there’s  direct  relationship  and  it  exists  at  different  levels  of 
intimacy,  professional  intimacy.  Online  you  can  sort  of  feign  that  and  l always  in  email  try  to 
use,  I learned  this  in  ’84.  ’85  when  I was  the  MBA  Chairman  dealing  with  the  MBA  students 
that  didn ’t  have  a class.  Always  I feel  it’s  good  to  use  warm  language.  So,  it’s  not,  the 
meeting  is  5,  it’s  Fd  love  to  have  a meeting  with  you  tomorrow.  It’s  to  keep  the  language  that 
you  would  never  in  direct  verbal  communication  because  it  would  be  artificial  but  it  warms  a 
cold  environment,  right?  But  I’d  like  to  find  a way  to  warm  the  environment  with  more  than 
the  use  of  exaggerated  language. 

Course  Context 

Online  Discussion  Rules 

The  findings  from  activity  analysis  suggested  the  there  was  a contradiction  embedded  in  the  rules  of 
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online  discussion,  that  is,  the  mandatory  vs  optional  participation  in  discussion  forum  It  was  observed  that 
when  online  participation  was  mandated,  the  majority  of  the  students  tended  to  post  for  visibility.  Their 
attentions  were  blocked  on  the  participation  grade  rather  than  meaningful  participation.  When  it  was  optional, 
there  were  few  participants.  Most  students  did  not  participate  and  attributed  it  to  the  reason  of  timing 
competitions.  When  asked  whether  an  online  discussion  should  be  made  mandatory  to  stimulate  more 
interactions  in  an  online  course,  one  participant  commented, 

“1  agree  with  it  as  well.  Blit  I don't  think  that's  enough.  If  participation  is  made  mandatory 
like  in  the  marketing  class  vs.  accounting  class,  true  that  most  people  will  post  on  time,  but 
more  people  post  their  comments  about  the  topic  instead  of  comments  about  others'  posting.  If 
it  is  optional,  I think  it  mostly  depends  on  how  busy  I am.  For  example,  I am  so  busy  with  the 
marketing  class  and  trying  to  catch  up  with  the  accounting  class  reading  that  it  is  very 
difficult  to  push  myself  to  get  involved  in  the  optional  online  discussion  in  the  accounting 
class.  ” 

The  findings  once  again  suggested  that  the  reality  of  online  students , their  extremely  busy  work  and 
study  life,  constrained  their  active  participation  in  online  learning  activities . 

Secondly,  findings  also  suggested  that  lacking  of  teacher’s  presence  seemed  to  be  one  of  the  reasons 
that  the  discussion  forum  was  not  attractive  to  students.  Some  students  indicated  that  instructor  needed  to  be 
involved  to  argument  and  redirect  the  discussion  to  make  it  more  interesting  and  refreshing  so  as  to  avoid  the 
comments  posted  by  the  students  reached  the  saturation  of  the  complexities  of  the  questions.  The  following 
quote  illustrated  the  point  that  the  presence  of  instructor  in  a mandatory  discussion  forum  could  facilitate 
meaningful  discourse  online. 

Ii  don 't  know  whether  I had  mentioned  my  last  course  I used  to  take  in  UM.  The  instructor 
had  kept  20%  grade  on  the  discussion  ...  what  she  had  done  was  that  she  said  she  would  read 
everybody's  post  and  then  she  had  said  that  you  cannot  be  just  posting  yes  or  no  such  answer, 
you  have  to  be  thoughtful  l...  there  has  to  be  someone  there  to  moderate  the  discussion  right. 

This  would  do  that  in  that  case. 

Course  structure  and  organization 

Almost  all  student  participants  agreed  on  that  “stay  with  the  schedule”  was  an  important  rule  for 
learning  online.  On  one  hand,  this  again  implied  the  competing  responsibilities  of  online  learner  which  caused 
the  difficulty  in  catching  up  with  the  schedule.  On  the  other  hand,  the  students  noted  that  any  unpredictable 
schedule  change  might  cause  disturbance  on  their  study  schedule  and  affect  their  learning  performance.  Many 
times  online  MBA  students  looked  for  regularity  in  course  design,  they  preferred  weekly  based  modular  format, 
clearly  specified  deadlines  and  regular  deliverables.  To  have  structured  course  plan  seemed  to  be  important  for 
them  to  plan  ahead  and  ensure  their  participating  in  an  online  course. 

Because  if  you  slow  done,  you  are  going  to  be  passed  up.  The  course  will  leave  you  behind 
especially  when  it  is  in  a hurry  pace.  That  is  something  to  think  about,  have  twelve  weeks.  You 
almost  jamming  everything,  you  are  doing  your  best  to  stay  up  with  this,  to  absorb  and  to  own 
the  knowledge  you  are  taking  in.  that  is  a tough  thing  to  think  about  it. 

I wish  there  were  more  frequent  assignments  in  this  class  since  that  would  insure  I'm  grasping 
the  material  incrementally  and  on  schedule.  Assignments  are  a couple  weeks  apart  and  cover 
content  from  several  chapters  so  it's  difficult  to  know  if  you're  understanding  the  content  as 
well  as  needed  since  the  assignments  and  quizzes  are  scheduled  so  far  apart. 

However,  the  findings  also  suggest  that  there  was  a tension  between  the  structure  and  the  flexibility 
of  course  design.  Overly  structured  online  course  might  cause  disturbance  in  students’  activities.  Within  overall 
structure,  the  threshold  of  flexibility  was  also  essential  because  this  was  the  point  that  they  chose  online 
program.  For  extmple,  several  days  time  frame  was  necessary  for  an  online  exam  so  that  it  can  accommodate 
individual  differences  on  their  work  schedule.  The  tension  of  structure  vs.  flexibility  was  well  illustrated  in  the 
quote  of  a student  from  Strategic  Marketing  Management  course. 
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I think  originally  there  were  a whole  lot  more  flexible  than  they  are  now.  Now,  you  know,  as 
you  get  into  the  MBA  classes  and  I think  as  they ’ve  gotten  more  experience  with  some  of  the 
classes,  it  seems  that  they 've  gotten  a little  more  rigid  on  you  having  specific  dates  and  times 
that  you  have  to  have  things  done.  In  at  least  some  of  my  classes,  I’ve  seen  those  to  be,  you 
know,  it’s  not  nearly  as  flexible  as  it  was  at  first.  You  have,  especially  with  a lot  of  group 
work.  You  know,  you’ll  have  a deadline  that’s  pretty  rigorous  and  in  order  to  get  going,  you 
have  to  start  ahead  of  time  planning  and,  you  know,  meeting  with  your  group  and  doing, 
coming  together  and  doing  different  deliverables  upfront  to  put  together  the  groundwork  to 
create  your  final  presentation,  so  I’ve  found  that  lately  it’s  gotten  more  just  like  a normal 
class.  But  you  still  do  know  upfront  what  you  have  to  do. 

Online  Community 

Analysis  of  interview  transcripts  revealed  that  the  students  who  participated  in  two  courses  had  a very 
weak  of  sense  of  belonging  to  an  online  class.  Instead,  the  sense  of  community  feeling  came  from  the  group 
they  worked  with.  For  example,  when  asked  whether  he  felt  a sense  of  community  in  this  course,  a participant 
stated:  “...  I haven’t  exchanged  notes  or  compare  thoughts  with  others  in  the  class.  To  me  the  class  is  [student 
name  1],  [student  name2]  and  the  professor.”  Another  participant  admitted  that  he  had  rarely  interacted  with  any 
students  other  than  the  group  members  and  teacher  in  this  course.  When  asked  whether  he  felt  a sense  of 
community,  he  stated,  ‘‘Not  really.  I think  maybe  with  the  professor  and  our  group,  but  not  necessarily  the  other 
groups  as  much.” 

Many  participants  indicated  the  sense  of  community  came  more  from  their  team  than  from  the  class. 
Sometimes  the  students  interacted  intensively  with  group  members,  however,  their  sense  of  community  didn’t 
go  beyond  the  group  community.  Two  students  commented, 

1 think  we  ’re  all  going  through  the  same  thing  in  the  course  and  in  reading  what  the  other 
classmates  are  thinking,  whether  it’s  different  or  the  same,  it  kind  of,  yeah,  it  forms  a kind  of 
community,  but  I think  the  major  community  that  we  do  form  is  with  our  own  team  members. 

Frankly,  not  in  this  course.  Even  in  other  courses,  the  learning  community  didn 't  develop 
beyond  the  group  members.  Only  in  Quant  course,  professors  used  to  hold  lectures  (literally) 
in  chat  rooms  at  specified  times.  That  was  the  only  real  learning  community  experience  so  far. 

Many  students  agreed  that  it  was  important  for  them  to  feel  a sense  of  belonging  to  an  online 
community  because  it  provided  them  with  socially  supportive  environment.  The  following  quote  demonstrated 
the  perceived  benefits  of  an  online  community. 

7 think  it  is  very  important  that  students  feel  they  are  in  it  together,  for  both  emotional  and 
academic  support.  It  is  not  easy  to  get  through  this  program  since  it  is  very  demanding.  And 
only  students  who  are  in  the  same  program  would  understand  the  difficulties. 

When  asked  to  give  an  example  of  a class  where  they  felt  a good  sense  of  community,  many 
participants  mentioned  a course  where  they  felt  a strong  sense  of  community  in  one  of  the  chat  rooms . An 
participant  commented, 

Yes,  I had  a decision  analysis  class  with  [instructor  name ] last  semester  that  had  a great 
sense  of  community.  We  had  scheduled  weekly  class  chat  sessions.  [Instructor]  instructed 
and  took  questions  from  the  class  on  the  material.  We  also  used  the  time  to  review 
assignments  and  ask  questions.  There  was  a lot  of  interaction  with  John,  with  our  TA,  and 
amongst  the  students.  It  did  a lot  for  my  confidence  in  learning  the  material  and 
understanding  the  process,  not  just  the  solution.  And  it  was  good  to  know  that  other  students 
had  the  same  questions/problems  I was  having. 

The  online  courses  featured  a weak  sense  of  belonging  to  a class  community  among  online  learners. 
However,  it  seemed  that  the  community  existed  at  the  small  group  level.  Even  so,  as  indicated  by  the  students, 
the  group  community  usually  discontinued  when  the  class  was  over. 
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Teamwork 

Majority  of  the  participants  indicated  that  in  online  learning  environment,  group  work  was  valuable 
because  the  group  brought  different  backgrounds  and  assets  of  the  people  together  to  share.  As  one  student 
commented, 

In  this  course  [name]  and  [name]  I am  working  with  we  worked  together  very  well.  Again  it  is 
neat  to  meet  these  GM  people  you  wouldn  ’t  meet  otherwise  because  GM  is  so  large , so 
diversified,  we  work  with  people  in  very  different  areas  that  have  very  different 
responsibilities  from  what  l have.  We  all  bring  different  backgrounds  and  different  assets  in 
these  groups.  It  is  a great  experience. 

Within  a team,  the  students  usually  divided  the  tasks  of  one  assignment  and  work  on  it,  and  then 
different  parts  were  compiled  together  as  one  group  deliverable.  The  requirement  for  the  coordination  on  the 
tasks  was  high,  which  was  in  conflict  with  the  efficiency-oriented  feature  of  teamwork  process.  Contradiction 
usually  occurred  when  long  meeting  time  was  spent  on  coordinating  different  opinions,  especially  when  the 
issues  were  not  related  to  the  critical  content  but  rather  to  the  task-related  issues.  One  participant  commented, 

The  teamwork  is  at  times  very  valuable  at  times  frustrating.  I will  take  the  barco  case  for 
example.  The  final  product  was  just  one  page.  A lot  of  team  issues  were  that  you  were  kind  of 
debating  among  yourselves  what  to  present  on  that  one  page.  It  wasn  ’t  technical.  It  was  more 
than  just  ttying  to  present  your  idea,  how  to  defend  your  own  ideas  ...  Sometimes  time  spends 
more  on  team  related  issues  rather  than  course  related  issues.  I was  much  preferred  to  have 
the  ability  to  have  my  own  response  to  work  with  a team  to  learn  and  then  with  our  own 
response. 

Though  students  valued  the  experiences  and  lessons  learned  from  dealing  with  different  personalities 
(identities)  in  a team  work,  such  alignment  of  different  personalities  (identities)  can  be  difficult  sometimes  that 
it  causes  resistances  on  collaboration  in  online  courses.  The  disturbances  within  teamwork  were  sometimes 
caused  by  the  conflict  or  inconsistent  views  on  quality  standards  about  the  group  product.  Specifically  this  kind 
of  issues  occurred  in  a team  where  majority  team  members  had  lower  standards  than  minority.  The  following 
quotes  represented  two  such  examples. 

Sometimes  I’ve  gotten , I’ve  told  somebody  that,  okay,  you  're  going  to  be  working  on  this  part 
of  the  assignment  and  they’ll  come  back  with  basically  just  crap  and,  you  know.  I’m  not  going 
to  turn  this  in,  so  I’m  left  trying  to  take  what  they’ve  given  me  and  revising  it  to  something 
that’s  at  least  legitimately  good  to  be  going  to  the  professor  and  that's  probably  an  advantage 
where  you,  if  you  were  in  class  and  you  were  able  to  get  a better  understanding  of  what 
people’s  capabilities  were  from  just  the  basic  interaction  that  you  would  have  in  class,  it 
would  really  help  out. 

There  are  problems  with  online  learning.  Participants  post  message  board  only  for  the 
purpose  of  assignments.  Very  few  teams  had  real  interactions  or  consistent  interactions  ... 

Whenever  we  dialed  up  in  the  teleconference,  and  found  out  that  there  was  not  much  to 
discuss  ( because  we  were  not  prepared).  Most  of  the  people  look  like  ...  OK,  when  is  this  due 
and  what  is  the  format,  then  we  started  dividing  our  task,  then  after  a while,  people  say,  sorry, 

I have  to  pick  up  my  children.  Then  the  meeting  is  over.  Very  few  people  will  sit  down  and  ask 
each  other,  what  you  think  of  this  question  ...  “ 

With  highly  efficient  and  task-oriented  teamwork,  conflicts  develop  between  high-performance,  task- 
oriented  goals  and  the  social-emotional  needs  of  the  participants.  Participants  indicated  that  their  teamwork  can 
mainly  satis  fy  the  task  needs  of  the  group  instead  of  social  need  due  their  tight  schedule. 

We  tried  not  to  waste  too  much  time  chatting  about  non -class  related  subjects.  Again,  it  all 
comes  down  to  time  spent  on  homework  vs.  time  spent  on  the  rest  of  your  life.  The  faster  we 
can  divide  the  work  and  reach  consensus  on  an  issue  the  better. 
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However,  the  participant  indeed  felt  lack  of  opportunities  to  socialize  with  team  members  made  it 
difficult  to  develop  good  group  dynamics.  In  traditional  classroom,  people  learn  from  body  language  and  facial 
expression.  In  online  environment,  the  loss  of  behavior  cues  created  barriers  for  students  to  really  get  to  know 
each  other’s  personalities.  The  participants  indicated  that  this  kind  of  “knowing”  is  important  because  it  could 
help  with  coping  with  different  personalities  to  create  a better  atmosphere. 

In  a physical  meeting , people  usually  come  a little  earlier.  For  teleconference,  people  always 
arrive  at  the  exact  time  or  only  one  or  two  minute  difference.  So  this  leaves  little  time  for 
socializing.  In  addition,  people  were  not  familiar  with  each  other  and  cannot  see  each  other’s 
body  language.  Even  people  can  speak  up,  they  are  mostly  superficial... Basically  we  don’t 
know  each  other.  We  don ’t  know  each  other's  personalities.  Some  people  are  more  aggressive 
and  some  are  more  conservative.  If  we  can  meet  face-to-face,  we  would  take  this  factor  into 
consideration.  However,  if  we  cannot  meet  face-to-face,  just  by  phone  or  internet,  it  takes 
longer  to  make  an  impression  on  that  person.  However,  this  course  only  lasts  ten  weeks. 

When  you  started  to  get  a sense  of  his  personality,  the  train  already  left  the  station. 

Conclusion 

Through  the  lens  of  activity  theory,  this  case  study  depicted  a rich  picture  of  online  MBA  courses  in  its 
activity  context.  The  findings  suggest  that  there  are  various  facets  of  an  online  course  that  affect  the 
performance  of  online  participants  as  well  as  the  success  of  the  course  overall.  In  reviewing  the  findings,  it  is 
not  difficult  to  conclude  that  it  is  important  to  design  a course  that  takes  into  the  consideration  of  learner’s 
social  context,  e.g.  competing  responsibilities,  the  commercialism  of  MBA  education  etc.  Understanding  these 
contextual  factors  is  important  for  us  to  decode  the  complexities  of  an  online  course,  the  priority  level  of  online 
students  and  their  performance  and  behavior  on  learning  tasks.  Such  considerations  could  result  in  the  deliberate 
selection  of  course  materials  that  do  not  over  teach  and  structure  meaningful  learning  experiences  that  really 
motivate  online  learners.  Secondarily,  the  findings  suggest  that  the  culture  of  an  online  course  seems  to  be 
temporal  and  it  might  be  difficult  to  facilitate  an  online  learning  community  within  a short  time  frame.  To 
facilitate  the  community  building  in  an  online  course,  there  needs  a structure  to  facilitate  the  building  of  a 
congenial  social  atmosphere  either  at  the  class  level  or  at  the  cohort  level,  e.g.  inter-group  activities,  social 
activities  among  cohort  level.  Finally,  there  seems  to  be  a fine  line  between  structure  and  flexibility.  Well- 
structured  and  planned  learning  environment  is  desirable  in  online  learning  environment.  However,  online 
course  design  also  needs  to  take  into  consideration  the  “flexibility”  factor  of  online  learning. 

As  a case  study,  the  generalization  of  the  results  from  the  context  of  this  study  is  limited.  However,  the 
strong  socio -cultural  themes  that  emerged  from  the  study  provide  a framework  for  assisting  distance  educators 
and  policy  makers  to  make  educational  policies  and  practices  for  providing  satisfactory  education  experiences 
for  various  participants  in  similar  online  learning  environment. 
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Abstract. 

This  study  examined  the  changes  in  the  level  of  intercultural  com/retence  of  high  school  students  that 
had  participated  in  a transnational  project  where  technology  was  utilized  to  exchange  information,  ideas,  and 
create  a fined  web-based  product.  Participants  of  the  study  included  162  tenth  grade  students  from  three 
different  high  schools  located  in  the  U.S.  and  Taiwan.  Data  was  collected  from  pre  and  post  self  assessment 
surveys  and  in-depth  interviews  with  selected  participants  were  conducted.  Results  show  that  American  students 
as  well  as  students  with  prior  travel  abroad  experience  had  higher  levels  of  intercultural  competence  compared 
to  Taiwanese  students  and  students  with  no  travel  experience.  It  was  discovered  that  prevalent  language 
barriers  had  prohibited  the  communication  and  collaboration  between  the  participating  students. 

Introduction 

Upon  entering  into  the  21st  century,  it  is  becoming  more  evident  of  how  increasingly  interdependent 
and  interconnected  we  are.  As  the  children  of  today  are  the  decision  makers  of  tomorrow,  it  is  imperative  that 
they  not  only  possess  the  knowledge  and  attitudes  for  becoming  responsible,  well-informed  citizens  of  the 
society  but  are  also  equipped  with  the  skills  and  capabilities  to  be  able  to  work  or  perform  in  a variety  of  diverse 
settings  that  would  call  for  interaction  with  people  from  different  cultures.  In  other  words,  our  students  need  to 
acquire  a global  perspective  (Ramler,  1991)  as  well  as  develop  a level  of  intercultural  competence  in  which  they 
have  “the  ability  to  interpret  intentional  communications  and  customs  in  cultures  different  from  one’s  own” 
(Bennett,  1999). 

According  to  Hanvey  (1979),  there  are  five  key  dimensions  to  a global  perspective;  perspective 
consciousness,  “state  of  the  planet”  awareness,  knowledge  of  global  dynamics,  awareness  of  human  choices, 
and  cross-cultural  awareness.  To  achieve  the  goal  of  cultivating  a global  perspective  in  our  next  generation,  we 
should  provide  students  with  learning  opportunities  that  will  help  enhance  their  awareness  and  understanding 
towards  intercultural  and  international  “human  relations,  critical  thinking,  social  sensitivity,  and  civic 
responsibility”  (Garcia,  1999;  Le  Roux,  2001).  In  addition,  Hughes-Wiener  (1988)  proposed  that 
internationalizing  the  curriculum  by  incorporating  cultural  understanding,  intercultural  sensitivity,  and  attention 
to  cultural  change  across  the  disciplines  will  help  students  develop  a greater  understanding  in  breadth  and  depth 
of  the  subject  being  learned. 

With  the  advent  of  information  communication  technology  (ICT),  especially  the  Internet,  valuable 
tools  and  resources  that  can  “bridge  gaps  in  international  communication  and  help  erase  cultural  and  social 
boundaries  between  countries”  (Lu,  2003;  Szente,  2003)  are  made  available.  These  rapid  advancements  in  ICT 
have  created  many  opportunities  for  teachers  and  students  to  be  able  to  acquire  immediate  access  to  the  world. 
Moreover,  curriculums  can  be  enriched  as  learning  is  contextualized  through  the  cultural  and  linguistic  diversity 
of  transnational  communities  that  are  built  through  the  Internet.  Hence,  we  have  seen  a growing  number  of 
classrooms  joining  online  learning  networks  and  communities  where  teachers  and  students  from  around  the 
world  are  able  to  get  together  to  work  on  collaborative  projects  or  carry  out  cross-cultural  email  exchanges. 
However,  not  all  outcomes  or  results  from  joining  online  learning  networks  or  email  exchange  projects  meet  the 
expectations  of  the  participants  (Lu,  2004).  Nevertheless,  despite  the  recent  trend  and  increasing  interest  in 
utilizing  technology  to  promote  cross-cultural  learning  and  collaboration,  few  studies  have  explored  the  impact 
of  these  learning  experiences  on  the  participants  in  terms  of  their  cultural  awareness,  attitudes  and  sensitivity. 
Therefore,  this  study  is  an  attempt  to  examine  the  impact  of  computer-mediated  intercultural  communication  on 
learners’  cultural  competence  that  includes  cultural  awareness,  understanding,  acceptance  and  appreciation  of 
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diversity  (Ward  & Ward,  2003). 


The  Intercultural  Communication  Over  the  Net  (ICON)  Project 

This  Intercultural  Communication  Over  the  Net  (ICON)  project  is  a transnational  collaborative  project 
between  one  high  school  in  the  U.S.  and  two  high  schools  in  Taiwan  that  was  facilitated  in  the  beginning  of  the 
year  2004  by  the  College  of  Education  at  the  University  of  Missouri  - Columbia,  USA  and  the  Graduate 
Institute  of  Education  in  National  Chiao  Tung  University,  Taiwan. 

Like  other  transnational  or  international  network  projects  via  telecommunications  technology,  such  as 
emailing  or  video  conferencing,  the  ICON  project  provided  the  opportunity  for  participating  teachers  and 
students  to  build  local-to-global,  cross-cultural  learning  through  the  interaction  and  collaboration  with  people 
from  other  nations  or  cultures.  Therefore,  the  ICON  project  researchers  conducted  an  exploratory  study 
examining  the  impact  of  learning  with  people  from  a different  culture  on  the  learner’s  level  of  intercultural 
competence. 


Method 


Sample 

The  sample  consists  of  students  (n=  162)  from  three  different  senior  high  schools  in  the  US  and 
Taiwan.  The  teacher  fromthe  high  school  in  St.  Louis,  Missouri  USA,  was  interested  in  partaking  in  a cross- 
cultural  exchange  project  and  therefore  agreed  to  have  her  students  in  all  four  of  the  classes  that  she  was 
currently  teaching,  totaling  64  students,  participate  in  the  ICON  project.  Also,  a teacher  from  Taiwan  had  her 
students  from  three  different  classes  in  two  different  high  schools  in  Taipei,  Taiwan,  totaling  98  students  join 
the  ICON  project  as  well. 

Materials 

The  ICON  project  researchers  were  responsible  for  creating  the  collaborative  activities  for  the 
participating  classes.  Many  factors  were  considered  in  the  design  and  development  of  the  interdisciplinary 
instructional  materials  and  instructional  tasks.  First  of  all,  in  order  to  prevent  the  ICON  project  from  becoming 
an  “add  on”  to  their  already  packed  curriculum,  the  expectations  of  the  teachers  and  the  overall  curriculum 
goals  had  to  be  taken  into  account  so  that  participation  in  the  project  will  be  seen  as  something  that  can  actually 
enhance  the  students  ’ learning  experience.  Therefore,  in  the  initial  stage  of  the  project,  the  researchers  had 
discussed  with  the  participating  teachers  what  their  expectations  of  the  ICON  project  were  and  how  they 
thought  it  would  fit  into  their  curriculum.  Information  gathered  from  the  discussions  helped  the  researchers 
determine  the  content  and  format  of  the  instructional  activities. 

Secondly,  it  was  essential  that  the  instructional  activities  focused  on  themes  that  introduced  the 
students  in  a systematic  way  to  different  cultural  traditions  and  norms  as  compared  to  their  own  so  to  foster 
intercultural  insight,  understanding  and  sensitivity.  However,  the  context  should  also  easily  relate  to  real  life 
experiences  so  that  the  students  are  encouraged  to  examine  and  reflect  on  their  own  perspectives,  attitudes  and 
cultural  background.  Therefore,  the  overall  theme  for  the  instructional  activities  was  “Going  abroad  to  study” 
and  the  sub-themes  consisted  of  “Classroom  culture”  and  “Scenic  spots  ”, 

Finally,  due  to  the  physical  distance  between  the  participants,  all  communication  and  collaboration 
were  to  be  done  through  the  Internet  utilizing  online  discussion  boards,  a computer  mediated  communication 
(CMC)  tool.  As  the  value  of  the  project  lies  in  the  quality  of  the  interaction  and  the  exchange  of  information  and 
ideas  between  the  students,  the  researchers  created  structured  discussion  board  topics  and  relevant  questions  to 
prompt  the  communication.  Therefore,  the  instructional  tasks  included  participation  in  structured  discussions 
with  an  assigned  partner;  one  discussion  board  for  each  of  the  two  sub-themes,  each  one  lasting  for  one  week, 
and  then  in  the  third  and  final  week,  the  paired  groups  create  a web  page  reporting  what  they  have  learned  from 
their  discussions. 

Instruments 

In  order  to  ensure  data  triangulation,  a combination  of  quantitative  and  qualitative  data  collection 
approaches  were  employed,  including:  (a)  pre  and  post  surveys,  (b)pre  and  post  interviews  with  the  teachers, 

(c)  post  interviews  with  selected  students,  and  (d)  student  performance  assessment  through  analysis  of  student 
artifacts . The  surveys  were  the  primary  source  of  data  as  the  follow-up  interviews  with  participating  students 
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provided  support  or  clarification  of  preliminary  findings. 

The  pre-/post  survey  instruments  included  the  researcher-developed  36-items  self-assessment  survey, 
three  open-ended  questions  and  a background  data  sheet  consisting  of  information,  such  as  gender,  cultural 
background,  travel  experience,  and  knowledge  of  other  languages,  were  also  collected  from  the  survey.  The  36 
items  were  adapted  from  existing  measures  of  multicultural  personality  and  awareness  and  cross-cultural 
sensitivity,  including  measures  developed  by  Cushner,  McClelland  & Safford  (2002),  D'Andrea,  Daniels,  & 
Heck  (1991),  Helfant  (1952),  Thurstone  (1931)  and  Van  der  Zee  & Van  Oudenhoven  (2000).  The  items  selected 
were  used  to  measure  the  intercultural  knowledge  and  competence  of  the  students.  It  also  aimed  to  understand 
students’  perception  and  attitudes  towards  other  cultures,  people  from  other  cultures,  and  international  relations. 


Design  and  procedure 

Implementation  of  the  project  started  in  early  March  of  the  year  2004  and  was  completed  in  early  May 
of  the  same  year.  In  this  study  the  effects  of  the  three-week  intervention  (i.e.  instructional  collaborative 
activities)  were  judged  by  the  difference  between  the  pre -survey  and  the  post-survey  results. 

The  intervention  consisted  of  two  aspects;  the  cultural  scenario  and  the  learning  partner.  The  cultural 
scenario  was  based  on  learning  about  either  American  or  Chinese  culture.  In  addition  to  learning  about  a 
specific  cultural  scenario,  the  students  were  divided  into  two  groups:  the  experimental  group  included 
intercultural  pairs  that  consisted  of  one  student  from  each  culture  (i.e.  one  American  student  and  one  Taiwanese 
student);  the  control  group  included  intracultural  pairs  that  consisted  of  students  from  the  same  culture  (i.e. 
either  all  American  students  or  all  Taiwanese  students). 

Students  were  asked  to  complete  required  activities  during  the  three-week  implementation  with  their 
learning  partner.  The  content  of  the  final  product  was  developed  during  the  pair  group  collaboration  and 
presented  via  web  pages. 

It  was  hypothesized  that  students  who  worked  with  a collaborative  learning  partner  from  another 
culture  will  develop  a greater  sense  of  cultural  awareness  and  sensitivity  than  those  working  with  a learning 
partner  from  the  same  culture.  This  is  based  on  the  belief  that  increased  cultural  knowledge  and  attitudes  are 
associated  with  greater  prior  contact  with  or  exposure  to  people  from  different  cultures.  Such  results  would 
demonstrate  the  significance  and  efficacy  for  participating  in  cross-cultural  collaborative  learning  projects  via 
the  Internet. 


Results 

Of  the  162  ICON  project  participants,  104  completed  both  the  pre -survey  and  post  survey.  As  stated 
earlier,  the  main  purpose  of  this  study  was  to  examine  the  impact  on  the  level  of  intercultural  competence  of 
students  that  had  participated  in  a collaborative  activity  with  a learning  partner  either  from  the  same  culture  or 
from  a different  culture.  In  Table  1 a crosstabulation  of  the  change  between  the  pre  and  post  survey  results  of 
the  American  and  Taiwanese  students  that  were  either  studying  about  their  own  culture  or  a different  culture 
with  a culturally  same  or  culturally  different  learning  partner  is  shown.  The  difference  between  the  pre  and  post 
survey  results  was  categorized  into  three  levels;  negative  impact  (pre  survey  results  > post  survey  results),  no 
impact  (pre  -survey  results  = post  survey  results),  and  positive  impact  (pre  survey  results  < post  survey  results). 

From  Table  1 it  shows  that  more  than  half  (60.5%)  of  the  American  students  self-assessed  that  the 
collaborative  project  had  an  overall  positive  impact  on  their  level  of  intercultural  competence.  However,  on  the 
other  hand,  only  a little  over  a third  (39%)  of  the  Taiwanese  students  thought  the  project  had  an  overall  positive 
impact. 


Table  1:  Crosstabulation  of  Pre-Post  Difference  * Learning  Partner  and  Cultural  Scenario  * Cultural 
Background 


Cultural  Background 

Learning  Partnerand  Cultural  Scenario 

Total 

Someone 
from  different 
culture, 
studying 
one's  own 
culture 

Someone 
from  different 
culture, 
studying  a 
different 
culture 

Someone 
from  same 
culture, 
studying  a 
different 
culture 
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America 

Pre-Post 

Negative 

Count 

2 

2 

3 

7 

n 

Difference 

impact 

% of  Total 

5.3% 

5.3% 

7.9% 

18.4% 

No  impact 

Count 

3 

1 

4 

8 

% of  Total 

7.9% 

2.6% 

10.5% 

21.1% 

Positive  impact 

Count 

3 

6 

14 

23 

% of  Total 

7.9% 

15.8% 

36.8% 

60.5% 

Total 

Count 

8 

9 

21 

38 

% of  Total 

21.1% 

23.7% 

55.3% 

100.0% 

Chinese 

Pre-Post 

Negative 

Count 

9 

8 

13 

30 

Difference 

impact 

% of  Total 

13.6% 

12.1% 

19.7% 

45.5% 

No  impact 

Count 

1 

3 

6 

10 

% of  Total 

1.5% 

4.5% 

9.1% 

15.2% 

Positive  impact 

Count 

4 

5 

17 

26 

% of  Total 

6.1% 

7.6% 

25.8% 

39.4% 

Total 

Count 

14 

16 

36 

66 

% of  Total 

21.2% 

24.2% 

54.5% 

100.0% 

An  independent  samples  T-  test  was  then  performed  to  check  if  there  was  a significant  difference 
between  the  American  and  Taiwanese  student  in  terms  of  the  impact  on  their  level  of  intercultural  competence 
after  participation  in  the  project.  Results  indicate  that  the  impact  of  participating  in  the  ICON  project  had  a 
significant  positive  difference  on  the  level  of  intercultural  competence  of  the  American  students  (M  = 1.61,  SD 
= 3.04)  than  the  Taiwanese  students  (M  = -.21,  SD  = 3.05),  t(102)  = 2.93,  p < .05. 

Therefore,  the  researchers  continued  to  examine  whether  student’s  prior  personal  experiences  such  as 
the  amount  of  travel  abroad  experience  had  any  influence  or  impact  on  their  level  of  intercultural  competence 
before  and  after  participation  in  the  project. 

In  Table  2 it  shows  the  pre -survey  results  of  American  and  Taiwanese  students  based  on  their  amount 
of  travel  abroad  experience.  The  researchers  categorized  the  pre -survey  results  into  three  levels  of  intercultural 
competence;  low  (total  survey  score  from  8-16),  moderate  (total  survey  score  from  17-25),  and  high  (total 
survey  score  from  26-32).  Also,  the  amount  of  travel  abroad  experience  was  categorized  into  three  levels  as 
well;  no  travel  experience  (has  never  traveled  before),  some  (have  traveled  1-3  times),  a lot  (have  traveled  at 
least  more  than  3 times). 

Results  show  that  in  the  case  of  having  no  travel  experience,  neither  American  nor  Taiwanese  students 
had  assessed  themselves  as  having  a high  level  of  intercultural  competence. 


Table  2:  Crosstabidation  of  Pre-survey  * Amount  of  Travel  Experience  * Cultural  Background 


Cultural  Background 

Amount  of  Travel  Experience 

Total 

None 

Some 
(traveled 
1-3  times) 

A lot 

(more  than 
3 times) 

American 

Pre-survey 

Low  intercultural 

Count 

2 

2 

1 

5 

competence 

% of  Total 

5.3% 

5.3% 

2.6% 

13.2% 

Medium  intercultural 

Count 

8 

8 

12 

28 

competence 

% of  Total 

21.1% 

21.1% 

31.6% 

73.7% 

High  intercultural 

Count 

0 

2 

3 

5 

competence 

% of  Total 

.0% 

5.3% 

7.9% 

13.2% 

Total 

Count 

10 

12 

16 

38 

% of  Total 

26.3% 

31.6% 

42.1% 

100.0% 

Chinese 

Pre-survey 

Low  intercultural 

Count 

6 

7 

3 

16 

competence 

% of  Total 

9.1% 

10.6% 

4.5% 

24.2% 

617 


Total 


Medium  intercultural 

Count 

18 

22 

8 

48 

competence 

% of  Total 

27.3% 

33.3% 

12.1% 

72.7% 

High  intercultural 

Count 

0.0% 

1 

1 

2 

competence 

% of  Total 

1.5% 

1.5% 

3.0% 

Count 

24 

30 

12 

66 

% of  Total 

36.4% 

45.5% 

18.2% 

100.0% 

Subsequently,  similar  to  Table  2,  in  Table  3 it  shows  the  post-survey  results.  Comparing  the  two 
Tables  2 and  3,  we  can  see  that  after  participation  in  the  ICON  project,  none  of  the  American  students  assessed 
themselves  as  having  a low  level  of  intercultural  competence  regardless  of  their  amount  of  travel  abroad 
experience.  In  addition,  the  number  of  American  students  with  “high  intercultural  competence”  increased  from 
5 to  8 students. 

On  the  contrary,  even  though  the  total  numbers  in  each  categorical  level  of  intercultural  competence 
for  the  Taiwanese  students  did  not  change  between  the  pre  and  post  survey,  the  numbers  in  each  of  the  cells 
however  did  change.  In  the  “No  travel  experience”  column,  not  only  did  the  number  of  low  level  intercultural 
competence  increased  from  6 to  10  Taiwanese  students  but  the  number  of  moderate  level  intercultural 
competence  also  decreased  from  18  to  14  Taiwanese  students  after  participation  in  the  project.  Then  again,  for 
Taiwanese  students  with  either  some  or  a lot  of  travel  abroad  experience,  the  numbers  in  the  low  intercultural 
competence  decreased  while  the  numbers  in  the  moderate  intercultural  competence  had  increased. 


Table  3:  Crosstabulation  of  Post-survey  * Amount  of  Travel  Experience  * Cultural  Background 


Cultural  Background 

Amount  of  Travel  Experience 

Total 

None 

Some 

(traveled  1- 
3 times) 

A lot 

(more  than 
3 times) 

American  Post-survey  Medium  Count 

intercultural  % of  Total 

competence 

7 

18.4% 

10 

26.3% 

13 

34.2% 

30 

78.9% 

High  intercultural  Count 

competence  % of  Total 

3 

7.9% 

2 

5.3% 

3 

7.9% 

8 

21.1% 

Total  Count 

% of  Total 

10 

26.3% 

12 

31.6% 

16 

42.1% 

38 

100.0% 

Chinese  Post-survey  Low  intercultural  Count 

competence  % of  Total 

10 

15.2% 

5 

7.6% 

1 

1.5% 

1 

24.2% 

Medium  Count 

intercultural  % of  Total 

competence 

14 

21.2% 

24 

36.4% 

10 

15.2% 

48 

72.7% 

High  intercultural  Count 

competence  % of  Total 

0 

.0% 

1 

1.5% 

1 

1.5% 

2 

3.0% 

Total  Count 

% of  Total 

24 

36.4% 

30 

45.5% 

12 

18.2% 

66 

100.0% 

Another  independent  samples  T-  test  was  performed  to  see  if  there  was  a significant  difference 
between  students  with  travel  experience  and  students  without  travel  experience  in  terms  of  the  impact  on  their 
level  of  intercultural  competence  after  participation  in  the  project.  Results  show  after  participation  in  the  ICON 
project,  students  with  prior  travel  abroad  experience  (M  = 25.24,  SD  = 4.31)  significantly  have  a higher  level  of 
intercultural  competence  compared  to  students  with  no  travel  experience  (M  = 23.59,  SD  = 3.73),  t(102)  = - 
2.02,  p < .05. 

Following  the  data  analysis  of  the  pre  and  post  survey  results,  the  researchers  proceeded  with 
reviewing  student  artifacts  and  conducted  in-depth  interviews  with  9 American  and  6 Taiwanese  students. 
Information  acquired  from  the  interviews  provided  valuable  insight  into  the  project’s  learning  experience  and 
the  students’  perspectives  in  terms  of  expectations,  limitations,  and  accomplishments. 


Discussion 

The  present  study  yielded  two  major  findings.  First,  prevalent  language  barriers  and  limitations 
seriously  inhibited  the  communication  and  collaboration  between  American  and  Taiwanese  students.  English 


618 


was  the  primary  language  used  in  the  project,  even  though  there  were  Chinese  versions  of  the  pre  and  post 
surveys  as  well  as  instructional  materials,  however  all  discussion  board  communication  and  development  of  the 
final  product  had  to  be  in  English.  Therefore,  being  English  as  a Foreign  Language  (EFL)  learner,  the 
Taiwanese  students  felt  deeply  challenged  and  frustrated  because  they  were  limited  by  their  lack  of  English 
proficiency.  One  Taiwanese  student  commented  in  her  interview  that  she  spent  more  time  looking  up  new 
vocabulary  in  the  dictionary  than  working  on  the  collaborative  activity.  On  the  other  hand,  some  American 
students  did  not  realize  that  language  barriers  were  the  main  reason  for  their  partners’  lack  of  response,  which 
had  made  them  feel  discouraged  at  the  time.  Referring  back  to  Table  1,  this  might  explain  why  there  were  such 
high  numbers  of  positive  impact  from  students  that  were  assigned  with  a learning  partner  from  the  same  culture 
because  language  barriers  may  not  have  been  as  a serious  problem  when  it  came  to  working  with  someone  from 
the  same  culture. 

Second,  the  experiences  of  exposure  to  different  people  and  culture,  such  as  traveling  abroad,  not  only 
enhances  a persons’  level  of  intercultural  competence,  but  it  also  increases  their  capability  and  motivation  to 
learn  and  develop  more  than  others  with  less  exposure  to  diverse  people  and  cultures.  In  other  words,  the  more  a 
person  experiences  culturally  different  people  and  things,  the  higher  their  level  of  intercultural  competence  is. 
Unfortunately,  not  all  people  have  the  opportunity  to  experience  or  become  exposed  to  diversity  due  to  their 
surrounding  environment  or  current  situation.  However,  rather  than  studying  about  a culture  from  reading  books 
or  watching  television,  the  project  researchers  believe  that  learning  with  and  from  a person  of  that  culture  is  the 
most  effective  and  motivating  way  to  learn  as  technology  plays  the  crucial  role  of  bridging  the  gaps  in  time  and 
distance  between  people  from  different  countries  and  cultures. 
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Abstract 

Intrinsic  motivation  can  be  predicted  from  participants  ’ perceptions  of  the  social  environment  and  the 
task  environment  (Ryan  & Deci,  2000)in  terms  of  control,  relatedness  and  competence.  To  determine  the  degree 
of  independence  of  these  factors  251  students  in  higher  vocational  education  (physiotherapy  and  hotel 
management)  indicated  the  extent  to  which  they  perceived  control,  relatedness,  and  competence  in  different 
types  of  education,  their  motivation  to  learn,  and  their  study  behaviors.  Principal  component  analysis  showed 
that  the  perceptions  of  control,  relatedness  and  competence  are  so  strongly  related  that  when  students  rate  one 
of  the  constructs  negatively,  such  as  their  perception  of  the  task  control,  they  will  also  rate  the  others 
(relatedness  and  perception  of  competence)  more  negatively.  These  results  were  confirmed  by  analyses  of  two 
additional  data  sets.  Consecptences  for  measurement  issues  and  motivational  science  are  discussed. 

Predicting  Intrinsic  Motivation 

In  the  past  two  decades  there  has  been  a strongly  renewed  interest  in  the  study  of  motivation  in  relation 
to  learning.  Simon  (1995),  for  example,  noted  that  it  “is  imperative  for  progress  in  instructional  methods  that  we 
deal  simultaneously  with  cognition  and  motivation  in  our  research. . . We  already  have  too  much  medicine  that 
is  (cognitively)  good  for  the  patient  - who  will  not  take  it  - and  medicine  that  patients  find  delicious  - but  that 
contributes  little  to  their  cognitive  abilities"(p.  508).  Pintrich  (2003)  described  this  as  a motivation  science 
coming  into  being. 

This  interest  has  much  to  do  with  a new  view  on  education.  Simons,  van  der  Linden,  and  Duffy  (2000) 
for  example  stress  new  instructional  methods  (in  their  words  “new  learning’')  such  as  independent  learning, 
discovery  learning,  experiential  learning,  self-directed  learning,  problem-oriented  education,  simulations,  and 
work -based  learning.  To  a large  extent,  these  methods  are  based  on  constructivism  in  which,  according  to 
Reiser  (2001),  learners  are  responsible  for  their  own  learning  process.  Such  self -regulated  learners  are 
motivated,  independent,  and  metacognitively  active  participants  in  their  own  learning  (e.g.,  Bastiaens  & 
Martens,  2000;  Dalgarno,  1998;  Duffy,  Herrington  & Oliver,  2000;  Lowyck  & Jonassen,  1993;  Pierce  & Jones, 
1998;  Wolters,  1998).  All  of  these  instructional  methods  hold  that  stimulating  motivation  is  crucial  in  learning, 
but  while  an  increasing  number  of  researchers  are  trying  to  link  instructional  strategies,  motivational  processes 
and  learning  outcomes  (Ellinger,  2004;  Garris,  Ahlers  & Driskell,  2002),  the  research  evidence  is  still 
“embryonic"  (Garris  et  al.,  p.  442). 

Motivation  is  not  an  easy  concept  to  define  because  it  is  related  to  many  partly  overlapping  theoretical 
constructs  (Norwich,  1999).  On  the  one  hand  it  is  be  seen  as  a relatively  stable  personality  trait  (e.g.,  Sheldon, 
Ryan,  & Reis,  1996),  while  on  the  other  hand  it  is  seen  to  vary  from  situation  to  situation  (e.g.,  Boekaerts  & 
Minnaert,  2003).  This  article  sees  motivation  as  the  latter  case  and  finds  its  roots  in  the  work  of  Ryan  and  Deci 
(2000)  who  distinguish  between  extrinsic  motivation,  which  refers  to  the  performance  of  an  activity  in  order  to 
attain  a certain  outcome,  and  intrinsic  motivation,  which  refers  to  doing  an  activity  for  the  inherent  satisfaction 
of  the  activity  itself.  The  effort  or  motivation  on  which  constructivist  learning  environments  try  to  rely  is 
typically  intrinsic  motivation,  with  its  associated  features  as  curiosity,  deep  level  learning,  explorative  behavior 
and  self  regulation  (Martens,  Gulikers,  & Bastiaens,  in  press).  Research  has  shown  that  intrinsically  motivated 
students  exhibit  study  behaviors  that  can  be  described  as  explorative,  reflective,  self-regulated,  and  aimed  at 
deep  level  processing  (e.g.,  Boekaerts  & Minnaert,  2003;  Martens  et  al,  in  press;  Ryan  & Deci,  2000). 

Ryan  and  Deci  developed  a model  to  explain  and  predict  the  persistence  of  intrinsic  motivation.  They 
state  : “...our  theory  of  motivation  does  not  concern  what  causes  intrinsic  motivation  (which  we  see  as  an 
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evolved  propensity;  Ryan  et  al,  1997);  rather  it  examines  what  conditions  sustain  versus  subdue  and  diminish, 
this  innate  propensity”  (p.  70).  This  approach  can  be  linked  to  evolutionary  psychology  and  can  be  found,  for 
example,  in  work  by  Bjorklund  and  Bering  (2002)  and  Bjorklund  and  Pellegrini  (2002).  In  their  view,  if  humans 
become  amotivated  there  is  a reason  for  this  in  their  perceived  social  and  physical  environment.  This  is  quite 
different  from  the  often  taken  developers’  viewpoint  that  it  is  the  developer  (of  education,  of  games,  of  ...)  who 
has  to  make  the  material  motivating  (e.g.,  the  ARCS  Model  of  Motivation  from  Keller  and  Suzuki,  1988).  In 
spite  of  this  theoretical  difference,  practical  advice  and  guidelines  derived  from  both  viewpoints  partly  overlap. 

Cognitive  Evaluation  Theory  (Ryan  and  Deci,  2000)  predicts  that  the  perception  of  certain  aspects  of 
the  social  and  task  environment  are  crucial  to  intrinsic  motivation.  These  perceptions  may  influence  each  other, 
but  exactly  how  is  unclear.  More  specifically,  a sense  of  relatedness,  control  or  competence  is  seen  to  be 
positively  correlated  with  intrinsic  motivation.  Various  authors  describe  these  aspects  as  predictors,  factors  or 
mediators.  If,  for  instance,  the  amount  of  control  is  varied,  then  the  perception  of  control  can  be  considered  as  a 
mediator  between  variation  in  control  and  intrinsic  motivation.  For  purposes  of  clarification,  this  article  will 
speak  of  the  perception  of  relatedness,  control  and  competence  as  predictors  since  they  can  be  seen  to  predict 
intrinsic  motivation. 

Cognitive  Evaluation  Theory  describes  stages  in  motivation,  varying  from  amotivation  via  introjection 
to  intrinsic  motivation  for  which  scales  have  been  developed.  Although  the  debate  is  still  going  on,  quite  some 
research  evidence  for  this  model  has  been  built.  Some  of  this  evidence  will  be  presented  here. 

Perception  of  control  has  been  shown  to  be  positively  correlated  with  intrinsic  motivation  (Enzle  & 
Anderson,  1993;  II  a id  re  & Reeve,  2003;  Nichols,  2004;  Pelletier,  Seguin -Levesque,  & Legault,  2002).  Raffini 
(1996)  stated  that  students’  need  for  a sense  of  autonomy  or  self-determination  significantly  influences  their 
intrinsic  motivation  to  learn  in  the  classroom  and  stresses  the  importance  of  building  a sense  of  autonomy  in 
students  by  providing  them  with  choices.  This  effect  has  also  been  reported  for  educational  software,  for 
example,  by  Cordova  and  Lepper  (1996)  and  Kinzie,  Sullivan  and  Berdel  (1988).  Iyengar  and  Lepper  (2000) 
also  found  that  too  much  choice  or  control  can  be  experienced.  Deci,  Koestner  and  Ryan  (1999)  performed  a 
meta-analysis  on  the  effects  of  extrinsic  rewards  - a form  of  external  control  (Henderlong  & Lepper,  2002; 
Norwich,  1999)  - on  intrinsic  motivation  and  found  that  many  forms  of  external  reward  undermine  intrinsic 
motivation. 

Perceived  competence  is  the  whole  complex  of  beliefs  about  one’s  own  competences  and  as  such  is 
highly  related  to  self-esteem,  the  evaluation  of  one’s  self-concept.  According  to  Harter  (1990),  perceived 
competence  is  an  important  psychological  mediator  of  achievement  behavior  and  motivation  among  children 
and  adolescents  in  the  academic  domain  and  has  often  been  demonstrated  to  affect  intrinsic  motivation.  In  a 
correlational  study,  children's  self-reported  perceptions  of  academic  competence  and  personal  control  were 
found  to  be  positively  related  to  their  intrinsic  interest  in  schoolwork  and  preference  for  challenging  school 
activities  (Boggiano,  Main,  & Katz,  1988).  Competence  can  be  perceived  through  praise,  through  comparisons 
with  other  students  or  other  indications  of  good  performance  or  through  meaningful  effort  (e.g.,  Henderlong  & 
Lepper,  2002).  Finally,  objective  mastery  praise  has  been  shown  to  be  better  than  social  comparisons  in 
affecting  motivation  (Henderlong,  Tomlinson,  & Stanton,  2004). 

Finally,  a sense  of  relatedness  (belongingness  or  connectedness  with  others)  has  quite  often  been 
demonstrated  (e.g.,  Ryan  & Deci,  2000;  see  also  Furrer  & Skinner,  2003  for  an  overview)  to  have  a positive 
impact  on  intrinsic  motivation,  and  thus  on  engagement  and  persistence.  Relatedness  is  characterized  by 
fulfillment  and  involvement  with  the  social  world.  This  social  aspect  affects  relatedness  by  creating  a climate  or 
culture  of  trust,  respect,  caring,  concern,  and  a sense  of  community  with  others.  In  a related  area  Kreijns  and 
Kirschner  (2004)  have  studied  the  role  of  this  social  interaction  in  collaborative  learning.  They  show  that  the 
existence  of  a sound  social  space  - the  network  of  social  relationships  amongst  the  group  members  embedded  in 
group  structures  of  norms  and  values,  rules  and  roles,  beliefs  and  ideals  - is  essential  for  reinforcing  social 
interaction.  A social  space  is  'sound'  if  it  is  characterized  by  affective  work  relationships,  strong  group 
cohesiveness,  trust,  respect  and  belonging,  satisfaction,  and  a strong  sense  of  community  (cf.  Rourke,  2000; 
Rovai,  2001). 

What  then  are  the  effects  of  intrinsic  motivation  on  learning?  Cordova  and  Lepper  (1996)  tried  to 
increase  children’s  intrinsic  motivation  in  educational  software.  As  predicted,  children  exposed  to 
motivationally  embellished  activities  displayed  higher  levels  of  intrinsic  motivation.  As  a result,  they  became 
more  deeply  involved  in  their  activities,  used  more  complex  operations,  and  learned  more  from  the  activities  in 
a fixed  period  of  time.  Another  effect  found  is  that  the  risk  of  drop-out  decreases  (Hardre  & Reeve,  2003; 
Vallerand,  et  al.,  1997).  Intrinsically  motivated  students  are  more  persistent  and  more  likely  to  achieve  set  goals 
(Curry,  Wagner  & Grothaus,  1990),  and  have  higher  levels  of  self-regulation  (Pintrich  & de  Groot,  1990)  than 
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those  who  are  not  intrinsically  motivated.  Intrinsically  motivated  adult  students  tend  to  exhibit  higher  subjective 
well-being  (Levesque,  Zuehlke,  Stanek,  & Ryan,  2004).  Low  intrinsic  motivation,  on  the  other  hand,  has  been 
shown  to  be  correlated  with  educational  self-handicapping,  avoidance  behavior,  loss  of  social  support  networks, 
and  passivity  (Thompson,  2004).  Overviews  (e.g.,  Ryan  & Deci)  indicate  that  intrinsically  motivated  students 
are  more  curious  and  engage  in  more  deep  level  learning,  an  effect  that  holds  true  for  students  of  all  age  groups 
(cf.  Bruinsma,  2003;  Turner  et  al,  1998;  Wolters  & Pintrich,  1998;).  It  is,  however,  not  necessarily  the  case  that 
more  extrinsically  motivated  students  always  do  less  (Ryan  & Deci,  2000).  Martens  et  al  (in  press)  found  that 
students  with  high  intrinsic  motivation  do  not  do  more  in  computer  based  learning  programs,  but  do  different 
things  (i.e.,  they  exhibit  more  exploration  behavior).  A compounding  problem  is  that  it  sometimes  is  impossible 
to  design  educational  tasks  that  are  intrinsically  motivating  to  all  students  (Kaufmann  & Husman,  2004).  In 
other  words,  we  need  to  know  more  about  how  motivation  affects  cognition.  This  is,  according  to  Pintrich 
(2003),  one  of  the  leading  questions  to  be  answered  by  the  "motivational  science’.  Thus,  although  it  is  clear  that 
perception  of  control,  relatedness  and  competence  are  related  to  intrinsic  motivation,  it  is  unclear  how  they  are 
interrelated.  Rarely  do  researchers  present  data  about  such  mediators  or  predictors  and  even  more  rarely  are 
these  data  presented  with  all  the  possible  mediators  together.  That  there  are  relations  is  evident,  since  all  these 
predictors  are  related  to  scales  measuring  intrinsic  motivation. 

For  any  science  the  exact  measurement  of  its  basic  constructs  is  crucial.  Unfortunately,  in  motivational 
research  there  are  four  important  problems  that  hinder  this,  namely. 

• It  is  unclear  how  control,  relatedness  and  competence  are  related.  Do  they  strengthen  each  other?  Do 
they  compensate  for  each  other? 

• Most  researchers  do  not  measure  these  three  predictors.  Reeve,  Nix  and  Hamm  (2003),  based  upon  an 
analysis  of  more  than  300  studies,  conclude  that  very  few  researchers  actually  investigate  the  impact  of 
what  may  be  called  autonomy  (or  lack  of  control  or  choice)  as  a possible  mediator  in  the  relation 
between  choice  and  intrinsic  motivation. 

• There  are  serious  measurement  and  definition  problems  such  as  what  the  exact  definition  is  of 
perceived  control.  Deci  and  Ryan  (1987),  for  example,  see  autonomy  or  the  absence  of  external  control 
as  a theoretical  concept  connoting  an  inner  endorsement  of  one's  own  actions  (origin,  personal 
causation,  internal  locus),  an  experience  during  that  action  of  high  flexibility  and  low  pressure 
(psychological  freedom),  and  a sense  that  one's  actions  are  truly  chosen  (perceived  choice).  Current 
theoretical  statements  treat  these  qualities  as  overlapping  and  mutually  supportive,  but  others  doubt 
this  (e.g.,  Reeve  et  al.,  2003). 

• Investigators  who  routinely  use  different  psychometric  measures  to  operationally  define  the  three 
predictors  often  rely  either  on  a single  item  predictor  (Boggiano  et  al.,  1993;  Eisenberger,  Rhoades,  & 
Cameron,  1999;  Overskeid  & Svartdal,  1996)  or  only  a pair  of  items  (Reeve  & Deci,  1996;  Thompson, 
Chaiken,  & Hazelwood  , 1993).  According  to  Reeve  et  al.  (2003)  “researchers  question  the  validity, 
internal  consistency,  and  conceptual  ambiguity  these  measures  generate.  This  limits  researchers' 
attempts  to  theoretically  understand  and  operationally  define  self-determination  as  an  ephemeral, 
situationally  sensitive,  and  statelike  experience  (i.e.,  state  self-determination).  In  contrast,  efforts  to 
assess  perceived  self-determination  as  an  enduring  characteristic  in  the  personality  (i.e.,  trait  self- 
determination)  have  been  deemed  psychometrically  sound.”  (p.  375).  Highly  reliable,  valid,  and 
educationally  useful  instruments  to  assess  trait  self-determination  include  the  Academic  Self- 
Regulation  Questionnaire  (Ryan  & Connell,  1989),  the  Causality  Orientations  Scale  (Deci  & Ryan, 
1985),  and  the  Academic  Motivation  Scale  (Vallerand  et  al.,  1992).  Unfortunately  as  soon  as  we  try  to 
more  measure  perceived  self-determination  as  based  on  the  context  or  situation,  it  is  a less  stable 
measure.  However,  this  is  exactly  what  happens  in  most  cases.  Based  on  adaptations  from  scales  used, 
most  researchers  develop  their  own  variations  that  are  specific  for  a certain  context . These  adapted 
scales  are  quite  often  used  in  correlational  research  and  covariance  structure  analysis  alternatively 
known  as  Structural  Equation  Modeling  (SEM). 

This  brings  us  to  the  main  research  question  of  this  article,  namely:  If  multiple  reliable  scales  to  measure  the 
perception  of  control,  relatedness  and  competence  in  different  educational  situations  are  constructed,  how  many 
factors  then  underlie  these  predictors?  An  explorative  analysis  investigated  the  number  of  factors  that  underlie 
the  predictors  focusing  not  on  questionnaire  construction,  but  on  the  connection  between  the  predictors  that  the 
scales  measure.  In  addition,  the  study  behavior  that  coincides  with  high  intrinsic  motivation  is  also  studied. 
Bruinsma  (2003),  Ryan  and  Deci  (2001),  Turner  et  al.  (1998)  and  Wolters  and  Pintrich  (1998)  all  found  that 
intrinsically  motivated  students  generally  learn  at  a deeper  level,  are  more  self -regulated  and  are  more 
communicative  than  those  who  are  not.  These  effects  appear  to  occur  together,  possibly  meaning  that  the 
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predictors  are  linked.  If,  for  instance,  high  motivation  coincides  with  study  behavior  in  which  the  learner  is 
more  communicative,  then  it  is  likely  that  the  predictors  are  strongly  related  since  it  is  quite  conceivable  that 
communication  is  related  to  a perception  of  relatedness. 

Method 


Participants 

Participants  were  251  full-time  undergraduate  students,  studying  Physiotherapy  or  Hotel  Management 
at  a Dutch  polytechnic.  They  were  in  the  second,  third  or  final  year  of  the  four  year  program  and  had  an  average 
age  of  20.8  years  ( SD  = 2.04).  Seventy  five  percent  of  the  participants  was  female.  Participation  (the  filling  in 
questionnaires)  was  voluntary  and  anonymous  with  a response  rate  of  80%. 

Procedure 

Five  distinct  educational  systems  were  distinguished  in  five  specific  courses  given  at  the  polytechnic,  namely: 

• Skills-based,  where  physiotherapy  students  were  involved  in  skills  training; 

• Problem  based  learning  (PBL),  where  physiotherapy  students  received  education  based  upon  specific 
physiotherapy  problem  cases  in  tutor  groups; 

• 4C/1D,  where  students  received  education  according  to  a competency  based  educational  approach  set 
up  following  the  Four-Component  Instructional  Design  (Van  Merrienboer,  1997); 

• Practice-based,  where  physiotherapy  students  were  involved  in  their  internship  period  at  the  end  of 
their  education;  and 

• Virtual  Hotel  School  (VHS),  where  students  from  the  Hotel  Management  School  work  together  in  a 
virtual  company. 

Instruments 

Four  scales  were  developed  / adapted  to  measure  perceived  control,  perceived  competence,  perceived 
relatedness  and  intrinsic  motivation.  Each  scale  contained  between  three  and  seven  items  (7 -point  Likert  scales). 
In  Table  1 the  scales  are  summarized  with  respect  to  their  size  and  reliabilities.  Though  generic  in  nature,  the 
scales  were  minimally  specified  to  suit  the  specific  educational  setting  in  which  they  were  administered  (i.e., 
substitution  of  the  word  problem  in  PBL  for  task  in  4C/ID).  Most  items  were  common  over  all  educational 
types.  Table  1 also  contains  examples  of  the  items  (translated  from  Dutch). 

Tablel.  Scale  Reliability 


Scale  and  example  of  item Cronbach’s  Alpha,  (number  of  items)  and  number  of  respondents 


Skills 

PBL 

4C/ID 

Practice 

VHS 

Perceived  control 

.65  (3) 

.60  (4) 

.65  (5) 

.57(3) 

.56(3) 

‘This  activity  was  mandatory’ 

n=65 

n=64 

n=50 

n=27 

n=36 

Perceived  competence 

.75  (3) 

.66  (3) 

.85  (5) 

.87(5) 

.64  (4) 

’I  think  I’m  good  at  this  activity’ 

n=65 

n=68 

n=53 

n=27 

n=34 

Perceived  relatedness 

.64  (6) 

.55  (3) 

.60  (5) 

.82(6) 

.71  (4) 

‘I  trust  my  peer  students’ 

n=62 

n=69 

n=51 

n=24 

n=34 

Intrinsic  motivation 

.84  (6) 

.87  (6) 

.75  (7) 

.91  (4) 

.96  (6) 

‘I  like  this  activity’ 

n=60 

n=65 

n=51 

n=26 

n=34 

Single  items  (7-point  Likert  scales)were  used  for  the  measurement  of  study  behavior,  and  thus  no 
reliability  scores  were  calculated.  These  items  measured  effort  invested  by  the  participant,  learning  aimed  at 
trying  to  understand  the  content,  trying  to  use  learning  content  in  practice,  having  discussed  the  content  with 
other  students,  concentration , ease  of  recall,  curiosity  about  content,  and  feeling  of  being  easily  distracted. 

Results 


The  Scales 

First,  the  bivariate  correlations  between  each  of  the  different  predictors  and  with  intrinsic  motivation 
were  determined  (see  Table  2).  Since  the  constructs  are  significantly  correlated  and  meet  with  all  parametric 
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criteria  necessary  to  calculate  such  correlations,  a principal  component  analysis  (PCA)  on  the  scales  was  then 
performed,  in  an  attempt  to  explain  as  much  variance  as  possible. 


Table  2.  Bivariate  Correlations  of  Predictors  and  Intrinsic  Motivation 


Perceived 

control 

Perceived 

competence 

Perceived 

relatedness 

Intrinsic 

motivation 

Perceived  control 

1 

.028 

.163* 

.174* 

Perceived  competence 

1 

.355** 

.411** 

Perceived  relatedness 

1 

.508** 

Intrinsic  motivation 

1 

*=p<.01;  **=p<.0001;  two-tailed,  n=250 


Principal  component  analysis  on  this  correlation  matrix  revealed  a 1 -component  solution  with  the  following 
loadings:  perceived  control  = .323;  perceived  competence  = .696;  perceived  relatedness  = .796,  and  intrinsic 
motivation  = .827. 


Table  3.  Toted  Variance  Explained  after  Principal  Component  Analysis 


Initial  Eigenvalue 

Extraction  SSL 

Component 

Total 

% of  Variance 

Cumulative  % 

Total 

% of  Variance 

Cumulative  % 

1 

1.907 

47.676 

47.676 

1.907 

47.676 

47.676 

2 

.980 

24.509 

72.185 

3 

.632 

15.798 

87.983 

4 

.481 

12.017 

100.000 

The  principal  components  analysis  shows  a 1-component  solution  with  the  criterion  eigenvalue  > 1, 
although  (with  a two  component  solution)  the  second  component  on  which  perceived  control  loads  does, 
explain  a substantial  part  of  the  variance.  Inspection  of  the  reliability  estimates  in  Table  1 indicates  that  a source 
of  error  variance  might  be  the  unreliability  of  the  perceived  control  scales.  In  other  words,  due  to  a certain 
amount  of  unreliability  of  the  measurement  of  one  of  the  four  variables,  the  correlations  between  these  variables 
may  be  underestimated.  This  can  be  rectified  by  making  use  of  a correction  of  attenuation  (e.g.,  Schmidt  & 
Hunter,  1996).  By  dividing  the  bivariate  correlation  by  the  root  of  the  weighted  reliability  coefficients  the 
estimated  correlation  coefficients  - if  both  variables  were  measured  with  perfect  reliability  - are  achieved.  These 
correlations  are  shown  in  Table  4. 


Table  4 Correlations  after  Correction  for  Attenuation 


Perceived  control 

Perceived 

competence 

Perceived 

relatedness 

Motivation 

Perceived  control 

1 

Perceived  competence 

.04 

1 

Perceived  relatedness 

.26 

.49 

1 

Motivation 

.24 

.51 

.69 

1 

Principal  component  analysis  based  on  this  correlation  matrix  again  yields  a 1-component  solution  with  the 
loadings;  perceived  control  = .384;  perceived  competence  = .732;  perceived  relatedness  = .874,  and  intrinsic 
motivation  = .878. 

The  correction  for  measurement  unreliability  shows  an  increase  in  the  explained  variance  (Table  5). 
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Table  5 Total  Variance  Explained  with  Principal  Component  Analysis  after  Correction  for  Attenuation 


Initial  Eigenvalue 

Extraction  SSL 

Component 

Total 

% of  Variance 

Cumulative  % 

Total 

% of  Variance 

Cumulative  % 

1 

2.218 

55.442 

55.442 

2.218 

55.442 

55.442 

2 

.968 

24.212 

79.655 

3 

.505 

12.614 

92.268 

4 

.309 

7.732 

100.000 

To  determine  whether  these  results  are  unique,  or  whether  there  may  be  other  corroborating  evidence,  two 
secondary  studies  were  carried  out.  First  this  approach  was  repeated  on  data  of  an  already  published  study 
(Ntoumanis,  2003).  Second,  new  research  with  a different  population  in  the  Netherlands  using  the  same 

instruments  was  carried  out.  These  results  will  be  reported  in  a forthcoming  article. 

Analyzing  the  Ntoumanis  (2003)  data  yields  the  same  picture  as  the  first  analysis  presented. 

Ntoumanis  uses  scales  to  predict  intrinsic  motivation  with  moderate  to  high  reliability  (ranging  from  .43  for 
autonomy  to  .87  for  intrinsic  motivation).  The  correlations,  standard  deviation  and  number  of  participants 
presented  by  Ntoumanis  were  entered  in  a matrix  as  input  for  PCA.  Before  correction  for  attenuation  a 
2-component  solution  was  found;  after  correction  a 1 -component  solution  was  found. 

Original  data  from  a second,  new  research  sample  was  also  analyzed.  This  sample  consisted  of  338 
higher  education  students  studying  in  the  Netherlands  in  full-time  higher  vocational  education  at  the  Maastricht 
School  for  Hotel  Management.  A small  majority  of  the  students  was  female  (60%)  and  had  an  average  age  of 
19.1  years.  The  students  filled  in  the  same  questionnaires  as  in  the  original  study.  All  scales  had  moderate  to 
high  reliability.  The  exact  coefficients  can  be  found  in  Appendix  2.  Again  PCA  resulted  in  a 1 -factor  solution, 
with  the  criterion  of  Eigenvalue  > 1. 

These  analyses  all  point  in  the  same  direction,  namely  that  if  the  unreliability  of  measurement  common 
for  scales  used  to  predict  motivational  processes,  scales  for  autonomy,  relatedness,  competence  and  intrinsic 
motivation  are  taken  into  account,  they  appear  to  measure  the  same  construct. 

Study  Behavior 

As  stated  earlier,  a second  aim  of  this  research  was  to  determine  the  impact  of  intrinsic  motivation  on  self-rated 
study  behavior.  Table  6 shows  that  intrinsic  motivation  appears  to  coincide  with  specific  study  behaviors. 


Table  6.  Correlations  between  Intrinsic  Motivation  and  Self  Reported  Study  Behavior 


Effort 

invested 

Learn  for 
under- 
standing 

Use  content 
in  practice 

Discuss 
content 
with  others 

Good 

concen- 

tration 

Easy  to 
remember 

Curious 

about 

content 

Feel  easily 
distracted 

intrinsic 

motivation 

.526 

.445 

.434 

.233 

.478 

.440 

.638 

-.240 

N 

250 

250 

223 

248 

248 

249 

247 

245 

All  correlations  are  significant  at  the  .001  level  (2-tailed) 


Students  with  high  intrinsic  motivation  exhibit  a higher  degree  of  effort,  perceive  the  learning 
materials  as  being  more  useful,  are  inclined  to  learn  more  for  understanding,  try  to  apply  what  they  learn  in  their 
practice,  and  appear  to  discuss  the  content  more  with  other  students.  They  report  having  better  concentration, 
more  curiosity  and  say  that  they  find  it  easier  to  remember  the  learning  content.  Finally,  they  report  feeling  less 
distracted  while  studying.  Again  this  combination  of  study  behaviors  accompanying  intrinsic  motivation  shows, 
as  predicted,  how  strongly  related  the  perceptions  are.  If,  for  example,  high  intrinsic  motivation  is  correlated 
with  more  inclination  to  discuss  and  communicate  what  has  been  learned  with  fellow  students,  this  will 
probably  be  linked  to  a more  positive  perception  of  relatedness. 

Discussion 

It  is  widely  accepted  (see  Ryan  and  Deci,  2000  for  an  overview)  that  perception  of  relatedness,  control 
(or  autonomy),  and  competence  predicts  intrinsic  motivation,  which  in  turn  predicts  study  behavior.  Less  clear 
is  how  these  three  predictors  are  related.  This  research  shows  that  they  are  so  strongly  correlated  that  in  fact 
they  appear  to  be  constituents  of  the  same  process.  Functionally,  they  form  one  single  factor.  This  means  that  a 
negative  perception  of  one  of  the  three  predictors  always  comes  jointly  with  a negative  perception  of  the  other 
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two.  In  plain  English:  if  you  feel  amotivated  because  someone  is  constantly  commenting  on  what  you  do  and 
telling  you  exactly  what  to  do  (thus  decreasing  your  perception  of  autonomy),  it  is  very  likely  that  you  will  not 
only  experience  a low  perception  of  relatedness  to  this  person  (i.e,  alienation),  but  that  you  also  will  not  feel 
very  competent  at  the  task  at  hand.  Or:  if  you  have  to  partake  in  a sport  activity  that  you  do  not  feel  that  you  are 
very  good  at  (i.e.,  a low  perception  of  competence),  you  will  probably  both  dislike  the  activity  (experience  a 
loss  of  intrinsic  motivation),  but  will  also  experience  a loss  of  relatedness  or  belongingness  with  your 
teammates.  In  other  words,  manipulations  intended  to  influence  any  one  of  the  three  aspects  - perceived 
relatedness,  competence  and  autonomy  - will  also  influence  the  other  two. 

This  study  also  shows  that  when  scales  to  measure  the  predictors  of  intrinsic  motivation  are  well 
constructed  (i.e.,  high  reliability)  one  can  rely  on  measuring  only  one  predictor,  since  the  other  predictors  all 
seem  to  be  part  of  the  same  mechanism(i.e.,  construct).  This  "mechanism’  also  explains  the  typical  reactions 
that  are  commonly  found  (e.g.,  Wolters  & Pintrich,  1998)  to  be  related  with  low  intrinsic  motivation.  These 
effects  were  replicated  in  this  study:  students  with  low  intrinsic  motivation  tend  to  be  less  inclined  to  interact 
with  their  peers  or  to  discuss  the  study  content,  phenomena  that  can  be  linked  to  lower  feeling  of  relatedness. 
This  lower  relatedness  can  be  interpreted  as  an  avoidance  tendency:  if  forced  to  join  a certain  a group  and 
engage  in  certain  unpleasant  activities  people  will  tend  to  want  to  leave  this  group. 

Some  critical  remarks  also  have  to  be  made.  First,  principal  component  analysis  is  a technique  where 
the  result  depends  on  many  elements.  The  same  holds  true  for  related  techniques  such  as  exploratory  or 
confirmatory  factor  analysis.  In  general,  discussions  on  the  number  of  factors  that  underlie  psychological 
measurements  are  difficult  and  lengthy.  It  took  a long  time  to  reach  agreement  about  the  ‘big  five’  factors  used 
to  map  personality  (e.g.,  Schmit  & Ryan,  1993)  as  well  as  on  the  factors  underlying  verbal  intelligence.  It  is, 
thus,  quite  likely  that  other  research,  both  exploratory  and  confirmatory,  with  other  data  might  lead  to  different 
factor  solutions,  even  though  this  article  reported  three  samples  that  all  pointed  in  the  same  direction.  Also,  the 
correction  for  attenuation  used,  only  yields  a hypothetical  estimate  of  the  correlation  that  would  appear  after  a 
perfect  measurement. 

Nevertheless  the  results  presented  in  this  study  provide  evidence  for  the  idea  that  there  appears  to  be  a 
connected  functional  mechanism  underlying  motivational  processes.  This  is  in  line  with  other  findings 
suggesting  (e.g.,  Ryan  & Deci,  2000;  Thompson,  2004)  that  suggest  that  diminished  intrinsic  motivation  comes 
along  with  a specific  but  broad  set  of  related  consequences,  varying  from  passivity,  to  lower  well-being,  less 
communication,  avoidance  behavior,  and  so  on.  This  mechanism  can  be  tentatively  termed  an  "amotivation 
module’. 


Acknowledgement 

The  authors  would  like  to  thank  S.  Braun  and  M.  Lemmens  from  the  Hogeschool  Zuyd  in  Heerlen,  The 

Netherlands,  for  their  help  and  cooperation  during  the  data  collection. 

References 

Bastiaens,  Th.,  & Martens,  R.  (2000).  Conditions  for  web-based  learning  with  real  events.  In  Abbey,  B.  (Ed.). 

Instructional  and  cognitive  impacts  of  web-based  education,  (pp.1-32).  Hershey/London:  Idea  Group 
Publishing. 

Bjorklund,  D.  F.,  & Bering,  J.  M.  (2002).  The  evolved  child.  Applying  evolutionary  developmental  psychology 
to  modern  schooling.  Learning  and  Individual  Differences , 12,  347-373. 

Bjorklund,  D.  F.,  & Pellegrini,  A.  D.  (2002).  The  origins  of  human  nature.  Evolutionary  developmental 
psychology.  Washington  DC:  American  Psychological  Association. 

Boekaerts,  M.  & Minnaert,  A.  (2003).  Assessment  of  students'  feeling  of  autonomy,  competence,  and  social 
relatedness:  A new  approach  to  measuring  the  quality  of  the  learning  process  through  self-  and  peer- 
assessment.  In  M.S.R.  Segers,  F.  J.  R.  C.  Dochy,  & E.  C.  Cascallar  (Eds.)  Optimizing  new  methods  of 
assessment:  in  search  of  quality  and  standards.  Dordecht,  NL:  Kluwer  Academic  Publishers. 

Boggiano,  A.  K.,  Flink,  C.,  Shields,  A.,  Seelbach,  A.,  & Barrett,  M.  (1993).  Use  of  techniques  promoting 
students ' self-determination:  Effects  on  students'  analytic  problem-solving  skills.  Motivation  and 
Emotion,  17,  319-336. 

Boggiano,  A.,  Main,  D.,  & Katz,  P.  (1988).  Children's  preference  for  challenge:  The  role  of  perceived 
competence  and  control.  Journal  of  Personality  and  Social  Psychology , 54,  134-141. 

Bruinsma,  M.,  (2003).  Leidt  hogere  motivatie  tot  betere  prestaties?  Motivatie,  informatieverwerking  en 
studievoortgang  in  het  hoger  onderwijs.  [Does  higher  motivation  result  in  higher  achievement? 


627 


Motivation,  cognitive  processing  and  achievement  in  higher  education].  Pedagogische  Studien,  80, 
226-238. 

Cordova,  D.,  & Lepper,  M.  (1996).  Intrinsic  motivation  and  the  process  of  learning:  Beneficial  effects  of 
contextualization,  personalization,  and  choice.  Journal  of  Educational  Psychology,  88,  715-730. 

Curry,  S.,  Wagner,  E.,  & Grothaus,  L.  (1990).  Intrinsic  and  extrinsic  motivation  for  smoking  cessation.  Journal 
of  Consulting  and  Clinical  Psychology,  58,  310-316. 

Dalgarno,  B.  (1998).  Choosing  learner  activities  for  specific  learning  outcomes:  A tool  for  constructivist 
computer  assisted  learning  design.  In  C.  McBeath  & R.  Atkinson  (Eds.),  Planning  for  Progress, 
Partnership  and  Profit.  Proceedings  EdTech'98.  Perth:  Australian  Society  for  Educational  Technology. 

Deci,  E.,  Koestner,  R.,  & Ryan,  R.  (1999).  A meta-analytic  review  of  experiments  examining  the  effects  of 
extrinsic  rewards  on  intrinsic  motivation.  Psychological  Bulletin,  125,  627-668. 

Deci,  E.  L.,  & Ryan,  R.  M.  (1985).  The  General  Causality  Orientations  Scale:  Self-determination  in  personality. 
Journal  of  Research  in  Personality,  19,  109-134. 

Deci,  E.  L.,&  Ryan,  R.  M.  (1987).  The  support  of  autonomy  and  the  control  of  behavior.  Journal  of  Personality 
and  Social  Psychology,  53,  1024-1037. 

Duffy,  T.  M.  Lowyck,  J.  & Jonassen,  D.  H.  (1993).  Designing  Environments  for  Constructive  Learning.  Berlin 
Springer-  V erslag . 

Eisenberger,  R.,  Rhoades,  L.,&  Cameron,  J.  (1999).  Does  pay  for  performance  increase  or  decrease  perceived 
self-determination  and  intrinsic  motivation?  Journal  of  Personality  and  Social  Psychology,  77,  1026- 
1040. 

Ellinger,  A.  D.  (2004).  The  concept  of  self-directed  learning  and  its  implications  for  Human  resource 
development.  Advances  in  Developing  Human  Resources,  6,  158-178. 

Enzle,  M.,  & Anderson,  S.  (1993).  Surveillant  intentions  and  intrinsic  motivation.  Journal  of  Personality  & 
Social  Psychology.  64,  257-266. 

Furrer,  C.,  & Skinner,  E.  (2003).  Sense  of  relatedness  as  a factor  in  Children’s  academic  engagement  and 
performance.  Journal  of  Educational  Psychology,  95,  148-162. 

Garris,  G.,  Ahlers,  R.,  & Driskell,  J.  (2002).  Games,  motivation,  and  learning.  Simulation  & Gaming,  33,  441- 
467. 

Harter,  S.  (1990).  Causes,  correlates  and  functional  role  of  global  self-worth:  A life  span  perspective.  In  J. 

Kolligan  & R.  Stenberg  (Eds.),  Perceptions  of  competence  and  incompetence  across  the  life  span.  New 
Haven,  CT:  Yale  University  Press. 

Hardre,  P.,  & Reeve,  J.  (2003).  A motivational  model  of  rural  students  intentions  to  persist  in,  versus  drop  out 
of,  high  school.  Journal  of  Educational  Psychology,  95,  347-356. 

Henderlong,  J.,  Tomlinson,  T.,  & Stanton,  P.  (2004).  Does  social  comparison  praise  undermine  children ’s 
intrinsic  motivation?  Paper  presented  at  the  AERA  2004  conference,  San  Diego. 

Henderlong,  J.,  & Lepper,  M.  (2002).  The  effects  of  praise  on  children's  intrinsic  motivation:  A review  and 
synthesis.  Psychological  Bulletin,  128,  774-795. 

Herrington,  J.  & Oliver,  R.  (2000).  An  instructional  design  framework  for  authentic  learning  environments. 
Educational  Technology  Research  and  Development,  48,  23-48. 

Iyengar,  S.,  & Lepper,  M.  (2000).  When  choice  is  demotivating:  Can  one  desire  too  much  of  a good  thing? 
Journal  of  Personality  & Social  Psychology,  79,  995-1006. 

Keller,  J.  M.,  & Suzuki,  K.  (1988).  Use  of  the  ARCS  motivation  model  in  courseware  design.  In  D.  H.  Jonassen 
(ED.)  Instructional  designs  for  microcomputer  courseware.  Hillsdale,  NJ:  Lawrence  Erlbaum. 

Kinzie,  M.,  Sullivan,  H.,  & Berdel,  R.  (1988).  Learner  control  and  achievement  in  science  computer-assisted 
instruction.  Journal  of  Educational  Psychology,  80,  299-303. 

Kaufman,  D.  F.,  & Husman,  J.  (2004).  Effects  of  time  perspective  on  student  motivation:  introduction  to  a 
special  issue.  Educational  Psychological  Review,  16,  1-7. 

Kreijns,  K.  & Kirschner,  P.  A.,  (2004).  Determining  sociability,  social  space  and  social  presence  in 
(a)synchronous  collaborating  teams.  Cyberpsychology  and  Behavior,  7(2),  155-172. 

Levesque,  Ch.,  Zuehlke,  A.  N.,  Stanek,  L.  R.,  & Ryan,  R.  M.  (2004).  Autonomy  and  competence  in  German 

and  American  university  students:  a comparative  study  based  on  self-determination  theory.  Journal  of 
Educational  Psychology,  96,  68-85. 

Martens,  R.,  Gulikers.  J.,  & Bastiaens,  Th.  (in  press.).  The  impact  of  intrinsic  motivation  on  e-learning  in 
authentic  computer  tasks.  Journal  of  Computer  Assisted  Learning. 

McAuley,  E.,  Duncan,  T.,  & Tammen,  V.  V.  (1987).  Psychometric  properties  of  the  intrinsic  Motivation 
Inventory  in  a competitive  sport  setting:  A confirmatory  factor  analysis.  Research  Quarterly  for 


628 


Exercise  and  Sport,  60, 48-58. 

Ntoumanis,  N.  (2003).  A self-determination  approach  to  the  understanding  of  motivation  in  physical  education. 
British  Journal  of  Educational  Psychology,  71,  225-242. 

Nichols,  J.  D.  (2004).  Empowerment  and  relationships:  a classroom  model  to  enhance  student  motivation. 

Paper  presented  at  the  AERA  2004  conference,  San  Diego. 

Norwich,  Br.  (1999).  Pupils’  reasons  for  learning  and  behaving  and  for  not  learning  and  behaving  in  English 
and  math  lessons  in  a secondary  school.  British  Journal  of  Educational  Psychology,  69,  547-569. 

Overskeid,  G.,  & Svartdal,  F.  (1996).  Effects  of  reward  on  subjective  autonomy  and  interest  when  initial  interest 
is  low.  The  Psychological  Record,  46,  319-331. 

Pelletier,  L.,  Seguin-Levesque,  C.,  & Legault,  L.,  (2002).  Pressure  from  above  and  pressure  from  below  as 

determinants  of  teachers'  motivation  and  teaching  behaviors.  Journal  of  Educational  Psychology,  94, 
186-196. 

Pierce,  J.  W.  & Jones,  B.  F.  (1998).  Problem  based  learning:  Learning  and  teaching  in  the  context  of  problems. 

In  Contextual  teaching  and  learning:  Preparing  Teachers  to  enhance  student  success  in  and  beyond 
school  (pp.  75-106).  Washington:  Eric  Clearinghouse  on  Teaching  & Teacher  Education  & Columbus, 
OH:  ERIC  Clearinghouse  on  Adult,  Career,  and  Vocational  Education 

Pintrich,  P.  R.,  & de  Groot,  E.V.  (1990).  Motivational  and  self-regulated  learning  components  of  classroom 
academic  performance.  Journal  of  Educational  Psychology,  82,  33-40. 

Pintrich.  P.  R.  (2003).  A motivational  science  perspective  on  the  role  of  student  motivation  in  learning  and 
teaching  contexts.  Journal  of  Educational  Psychology,  95,  667-687. 

Raffini,  J.  (1996).  150  ways  to  increase  intrinsic  motivation  in  the  classroom.  Boston,  MA:  Allyn  & Bacon. 

Reeve,  J.,  & Deci,  E.  L.  (1996).  Elements  within  the  competitive  situation  that  affect  intrinsic  motivation. 
Personality  and  Social  Psychology  Bulletin,  22,  24-33. 

Reeve,  J.,  Nix,  G.,  & Hamm,  D.  (2003).  Testing  models  of  the  experience  of  self-determination  in  intrinsic 
motivation  and  the  conundrum  of  choice.  Journal  of  Educational  Psychology,  95,  375-392. 

Rourke,  L.  (2000).  Operationalizing  social  interaction  in  computer  conferencing.  In  Proceedings  of  the  16th 
annual  conference  of  the  Canadian  Association  for  Distance  Education.  Quebec  City,  Canada. 
Retrieved  February  2,  2003  from  the  World  Wide  Web: 
http://www.ulaval.ca/aced2000cade/english/proceedings.html . 

Rovai,  A.  P.  (2001).  Classroom  community  at  a distance:  A comparative  analysis  of  two  ALN  -based  university 
programs.  Internet  and  Higher  Education,  4{ 2001),  105-118. 

Ryan,  R.  M.,  Koestner,  R.,  & Deci,  E.  L.  (1991).  Varied  forms  of  persistence:  When  free-choice  behavior  is  not 
intrinsically  motivated.  Motivation  and  Emotion,  15,  185-205. 

Ryan,  R.  M.,  Mims,  V.,  & Koestner,  R.  (1983).  Relation  of  reward  contingency  and  interpersonal  context  to 
intrinsic  motivation:  A review  and  test  using  cognitive  evaluation  theory.  Journal  of  Personality  and 
Social  Psychology,  45,  736-750. 

Ryan,  R.M.,  & Deci,  E.L.  (2000).  Self-determination  theory  and  the  facilitation  of  intrinsic  motivation,  social 
development,  and  well  being.  American  Psychologist,  55,  68-78. 

Ryan,  R.  M.,&  Connell,  J.  P.  (1989).  Perceived  locus  of  causality  and  internalization:  Examining  reasons  for 
acting  in  two  domains.  Journal  of  Personality  and  Social  Psychology,  57,  749-761. 

Schmit,  M.  J.  & Ryan,  A M.  (1993).  The  Big  Five  in  personnel  selection:  Factor  structure  in  applicant  and 
nonapplicant  populations.  Journal  of  Applied  Psychology,  78,  966-974. 

Schmidt,  F.  L.,  & Hunter,  J.  E.  (1996).  Measurement  error  in  psychological  research:  Lessons  from  26  research 
scenarios.  Psychological  Methods.  1,  199-223 

Schraw,  G.,  Flowerday,  T.,&  Reisetter,  M.  F.  (1998).  The  role  of  choice  in  reader  engagement.  Journal  of 
Educational  Psychology,  90,  705-714. 

Sheldon,  K.  M.,  Ryan,  R.  M.,  & Reis,  H.  (1996).  What  makes  for  a good  day?  Competence  and  autonomy  in  the 
day  and  in  the  person.  Personality  and  Social  Psychology  Bulletin,  22,  1270-1279. 

Simon,  H.  A.  (1995).  The  information-processing  theory  of  mind.  American  Psychologist,  50,  507-508. 

Simons,  R-J.,  van  der  Linden,  J.,  & Duffy,  T.  (2000).  New  learning.  Dordrecht/Boston:  Kluwer  Academic 
Publishers. 

Thomas,  S.,  & Oldfather,  P.  (1997).  Intrinsic  motivations,  literacy,  and  assessment  practices:  “That's  my  grade. 
That's  me.”  Educational  Psychologist,  32,  107-123. 

Thompson,  E.  P.,  Chaiken,  S.,  & Hazelwood,  J.  D.  (1993).  Need  for  cognition  and  desire  for  control  as 
moderators  of  extrinsic  reward  effects:  A person  x situation  approach  to  the  study  of  intrinsic 
motivation.  Journal  of  Personality  and  Social  Psychology,  64,  987-999. 


629 


Thompson,  T.  (2004).  Failure -avoidance:  parenting,  the  achievement  environment  of  the  home  and  strategies 
for  reduction.  Learning  and  Instruction,  14,  3-27 . 

Turner,  J.  C.,  Meyer,  D.  K.,  Cox,  K.  E.,  Logan.  C.M,  DiCintio,  M.,  & Thomas,  C.  T.  (1998).  Creating  contexts 
for  involvement  in  mathematics.  Journal  of  Educational  Psychology,  90,  730-745. 

Vallerand,  R.,  Fortier,  M.  S.,  & Guay,  F.  (1997).  Self-determination  and  persistence  in  areal-life  setting: 
Toward  a motivational  model  of  high  school  dropout. 

Journal  of  Personality  & Social  Psychology  72,  1161-1 176. 

Wolters,  C.  A.,  & Pintrich,  P.  R.  (1998).  Contextual  differences  in  student  motivation  and  self-regulated 
learning  in  mathematics,  English  and  social  studies  classrooms.  Instructional  Science,  26,  21-41 . 

Zuckerman,  M.,  Porac,  J.,  Lathin,  D.,  Smith,  R.,  & Deci,  E.  L.  (1978).  On  the  importance  of  self-determination 
for  intrinsically -motivated  behavior.  Personality  and  Social  Psychology  Bulletin,  4,  443-446. 


630 


Effects  of  Instructional  Events  in  Computer-Based  Instruction 

Florence  Martin 
James  Klein 
Howard  Sullivan 
Arizona  State  University 

Introduction 

Forty  years  ago,  Robert  Gagne  published  the  first  edition  of  his  book  The  Conditions  of  Learning 
(1965)  in  which  he  proposed  nine  events  of  instruction  that  provide  a sequence  for  organizing  a lesson.  These 
events  remain  the  foundation  of  current  instructional  design  practice  (Reiser,  2002;  Richey,  2000).  They 
represent  desirable  conditions  in  an  instructional  program  and  increase  the  probability  of  successful  learner 
achievement  in  the  program  (Gagne,  1965,  1985,  1988;  Gagne,  Briggs  & Wager,  1992).  Other  authors  cite 
similar  elements  of  instruction  that  promote  student  learning  from  an  instructional  program  (Dick  & Carey, 
1996;  Sullivan  & Higgins,  1983). 

Gagne  (1985)  defined  instruction  as  “a  set  of  deliberately  planned  external  events  designed  to  support 
the  process  of  learning.”  He  noted  that  a designer  or  instructor  controls  these  external  events,  and  that  learners 
control  their  own  internal  learning  processes.  His  external  events  of  instruction  were  conditions  for  facilitating 
effective  learning  processes  in  students. 

The  individual  events  that  Gagne  incorporated  into  his  model  have  been  the  subject  of  a substantial 
body  of  research.  However,  many  of  these  events  may  produce  a much  different  effect  when  they  are  studied 
individually  than  when  they  are  combined  into  a more  complete  set  that  incorporates  most  or  all  of  Gagne’s 
nine  events.  As  Hannafin  (1987)  noted,  some  design  strategies  may  have  positive  effects  when  used  in  isolation 
that  are  diminished  or  negated  when  these  strategies  are  used  in  combination  with  more  powerful  techniques. 

A research  design  that  incorporates  most  of  Gagne’s  events  of  instruction  into  a quite  complete  version 
of  an  instructional  program,  then  systematically  deletes  selected  events  from  other  versions,  has  the  potential  to 
identify  the  events  that  are  most  powerful  in  promoting  student  learning.  That  type  of  design  was  used  in  the 
present  research.  The  events  from  Gagne’s  model  that  were  directly  incorporated  into  the  study  were  objectives, 
information,  examples,  practice  with  feedback  and  review.  The  research  literature  on  each  of  these  events  is 
briefly  reviewed  below. 

Objectives 

An  instructional  objective  is  a statement  that  describes  an  intended  outcome  of  instruction  (Mager, 
1962).  According  to  Ausubel  (1968)  stating  an  objective  at  the  beginning  of  instruction  will  help  the  individual 
learners  to  structure  their  own  learning.  Reiser  and  Dick  (1996)  state  that,  ‘‘At  a fairly  early  stage,  learners 
should  be  informed  of  what  it  is  that  they  are  going  to  be  able  to  do  when  they  finish  the  instructional  process. 
By  knowing  what  will  be  expected  of  them,  learners  may  be  better  able  to  guide  themselves  through  that 
process”  (p.48). 

Some  researchers  have  found  that  instructional  objectives  improve  learning.  Kaplan  and  Simmons 
(1974)  reported  that  performance  on  information  relevant  to  an  objective  was  high  when  instructional  objectives 
were  used  as  orienting  stimuli  or  as  a summary/review  upon  prose  learning.  Staley  (1978)  found  that  the 
provision  of  objectives  facilitated  learning,  but  that  presenting  objectives  by  subsets  had  no  advantage  over 
presenting  the  entire  set  at  once.  Research  on  effectiveness  of  objectives  in  computer-based  cooperative 
learning  indicated  that  students  who  received  instructional  objectives  performed  significantly  better  on  posttest 
items  than  students  who  received  either  advance  organizers  or  no  orienting  activities  (Klein  & Cavalier,  1999). 
Studies  have  reported  that  objectives  enhance  learning  of  relevant  content,  but  provide  less  assistance  for 
incidental  learning.  (Kaplan  & Simmons,  1974;  Morse  & Tillman,  1972;  Rothkopf  & Kaplan,  1972).  Research 
has  also  indicated  that  inclusion  of  objectives  resulted  in  more  positive  student  attitudes  (Staley,  1978). 

Some  researchers  have  found  that  objectives  do  not  produce  a significant  difference  in  learning  (Filan 
& Gerlach,  1979;  Hartley  & Davis,  1976).  Hannafin  (1987)  found  that,  when  computer-based  instruction  was 
systematically  designed,  the  presence  of  objectives  did  not  make  a difference  but  that  it  did  influence 
performance  in  lessons  that  were  not  well  designed.  Research  has  also  indicated  that  the  benefits  of  objectives 
are  reduced  when  a more  powerful  instructional  element  such  as  practice  is  included  in  computer-based  lessons 
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(Hannafin,  1987;  Hannafin,  Philips,  Rieber  & Garhart,  1987;  Philips,  Hannafin  & Tripp,  1988). 

Information 

A significant  part  of  the  instructional  process  involves  presenting  students  with  the  necessary 
information  for  learning  (Reiser  & Dick,  1996).  All  models  of  direct  instruction  include  presenting  information 
to  students.  Gagne  (1985)  stresses  the  importance  of  emphasizing  the  information  presented  to  the  learners.  In 
his  nine  events  he  mentions  presenting  the  stimulus  or  content  where  information  is  presented  to  the  learner. 
Distinctive  features  of  what  is  to  be  learned  should  be  emphasized  or  highlighted  when  the  information  is 
presented  (Gagne,  1985).  Content  presented  should  be  chunked  and  organized  meaningfully.  (Kruse  & Kevin, 
1999). 

Practice  and  Feedback 

Practice  is  defined  as  the  event  of  instruction  provided  to  learners  after  they  have  been  given 
information  required  to  master  an  objective  (Gagne,  1985).  Practice  involves  eliciting  performance  from 
learners.  It  provides  an  opportunity  for  learners  to  confirm  their  correct  understanding,  and  the  repetition  also 
increases  the  likelihood  of  retention  (Kruse  & Kevin,  1999).  Practice  is  effective  when  it  is  aligned  with  the 
assessment  in  the  form  of  a posttest  and  with  the  skills,  knowledge  and  attitudes  reflected  in  the  objectives 
(Reiser  & Dick,  1996). 

Researchers  have  found  that  practice  has  a significant  effect  on  performance.  Hannafin  (1987)  reported 
a significant  difference  between  practiced  and  non-practiced  items  on  the  learning  of  cued  and  uncued 
information  presented  via  computer-based  instruction.  Phillips  et  al.  (1988)  found  a significant  difference 
favoring  practice  over  no  practice  in  an  interactive  video  in  which  practice  items  were  embedded  questions. 
Hannafin  et  al.  (1987)  noted  that  practice  effects  were  more  pronounced  for  facts  than  for  application  items  in 
computer-based  instruction.  Participants  who  received  intellectual  skills  practice  in  a cooperative  learning 
environment  performed  significantly  better  than  those  who  received  verbal  information  practice  (Klein  & 
Pridemore,  1994). 

Practice  provides  an  opportunity  for  feedback  that  confirms  the  student’s  answer  as  being  correct  or 
indicates  that  is  incorrect.  This  feedback  strengthens  the  probability  of  correct  responses  and  reduces  the 
probability  of  subsequent  incorrect  responses  (Philips  et  al.,  1988).  Simple  forms  of  feedback  are  effective  when 
learners  are  able  to  answer  items  correctly.  More  elaborate  forms  such  as  providing  and  explaining  the  correct 
answer  and  explaining  why  a wrong  answer  is  incorrect  are  helpful  when  learners  answer  incorrectly  (Kulhavy, 
1977).  Simple  forms  of  feedback  are  most  effective  for  simple  verbatim  and  verbal  information  types  of 
learning  (Kulhavy,  White,  Topp,  Chan  & Adams,  1985). 

Examples 

Examples  are  verbal  or  graphical  information  that  provides  additional  clarification  of  rules  or 
information  presented  to  learners.  Kruse  and  Kevin  (1999)  include  examples,  non-examples,  graphical 
representation  and  analogies  as  guidance  strategies  that  can  be  used  to  further  clarify  new  content  that  is 
presented. 

Few  studies  have  been  conducted  to  examine  effects  of  examples  in  a graphical  representation  form. 
Sullivan  and  Maher  (1982)  found  a significant  difference  favoring  the  use  of  imagery  over  no  imagery  in  prose 
learning  by  intermediate  grade  students.  Walcyzk  and  Hall  (1989)  reported  a significant  difference  for 
participants  who  received  examples  over  those  who  did  not  in  comprehension  assessments.  Freitag  and 
Sullivan  (1995)  found  that  adults  who  received  examples  in  a training  program  significantly  outperformed  those 
who  did  not.  A considerable  amount  of  research  has  been  conducted  recently  on  the  effects  of  worked  examples 
as  an  instructional  aid  (Atkinson,  Catrambone  & Merrill,  2003;  Atkinson,  Renkl  & Merrill,  2003;  Renkl,  Stark 
& Gruber,  1998). 

Review 

The  review  process  typically  provides  an  outline  of  the  key  information  that  was  presented  to  learners. 
It  is  intended  to  reinforce  learning,  at  the  end  of  the  instruction,  often  just  before  students  are  tested.  Reiser  and 
Dick  (1996)  cite  the  value  of  reviews  to  bring  closure  to  instruction  and  to  help  reinforce  the  skills  and 
knowledge  students  should  have  acquired. 

Research  has  suggested  that  reviews  benefit  learning  of  incidental  material  because  instructional 
stimuli  are  introduced  after  the  content  has  been  presented  and  initially  processed  (Kaplan  & Simmons,  1974). 
The  use  of  reviews  to  summarize  salient  information  has  been  shown  to  enhance  learning  (Hartley  & Davis, 
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1976).  In  studies  on  prose  learning,  reviews  of  relevant  information  yielded  significantly  better  performance 
than  when  the  information  was  presented  without  review  (Bruning,  1968). 


Purpose  of  Current  Study 

Many  of  the  studies  reported  above  were  conducted  to  examine  the  effect  of  a single  instructional 
event.  In  general,  these  studies  found  that  the  presence  of  the  event  under  investigation  resulted  in  a positive 
effect  on  student  learning.  It  was  also  noted,  however,  that  the  effects  of  some  of  these  events  may  be  reduced 
considerably  when  they  are  combined  with  other  events  into  a more  complete  and  generally  more  appropriate 
program  of  instruction. 

The  purpose  of  this  study  was  to  investigate  effects  of  several  of  Gagne’s  events  of  instruction  when 
they  were  combined  in  a systematic  manner  with  other  events  from  the  Gagne  set.  One  event,  information,  was 
constant  across  all  of  the  program  versions  in  the  study  because  information  is  a crucial  element  of  instruction 
that  cannot  sensibly  be  deleted  from  it.  The  other  events  of  instruction  investigated  in  the  study  - objectives, 
practice  with  feedback,  examples  and  reviews  were  combined  into  six  different  versions  of  an  instructional 
program  in  a manner  that  permitted  investigation  of  the  effectiveness  of  the  program  when  each  event  was 
present  and  when  it  was  absent. 

The  six  different  versions  of  the  instructional  program  were  as  follows: 

1.  Full  program  (Information  + Objectives  + Practice  with  Feedback  + Examples  + Review) 

2.  Program  without  Objectives  (Information  + Practice  with  Feedback  + Examples  + Review) 

3.  Program  without  Examples  (Information  + Objectives  + Practice  with  Feedback  + Review) 

4.  Program  without  Practice  (Information  +Objectives  + Examples  + Review) 

5.  Program  without  Review  (Information  + Objectives  + Practice  with  Feedback  + Examples) 

6.  Lean  program  (Information  Only) 

The  primary  research  questions  for  this  study  are  listed  below. 

1.  Which  of  Gagne’s  events  of  instruction  investigated  in  the  study  significantly  affect  student 
achievement? 

2.  Which  of  Gagne’s  events  of  instruction  investigated  in  the  study  significantly  affect  student  attitudes? 
The  researchers  anticipated  that  the  four  versions  of  program  that  included  practice  would  have  a positive 

effect  on  achievement  partly  because  of  the  consistently  favorable  effects  found  for  practice  in  other  research 
and  partly  because  of  our  own  beliefs  about  its  importance.  Whether  positive  effects  would  be  obtained  for  the 
other  variables  when  they  were  combined  with  practice  was  unclear  prior  to  the  study. 

Method 


Participants 

Participants  were  256  freshman  and  sophomore  undergraduate  students  enrolled  in  a computer  literacy 
course  at  a large  Southwestern  University.  The  students  enrolled  in  this  course  had  varied  background 
knowledge  on  computers  and  were  from  different  majors  including  education,  communication,  journalism  and 
others. 

Materials 

Six  different  versions  of  a computer-based  lesson  on  the  topic  Input,  Processing,  Storage  and  Output 
of  a Computer  (IPSO)  were  developed  using  Dreamweaver.  IPSO  explains  the  primary  operations  of  the 
computer.  An  introduction  section  was  included  before  the  primary  operations  were  explained  in  detail.  This 
section  introduced  what  a computer  is  and  classified  it  based  on  size,  power  and  generation.  It  also  explained 
the  IPSO  cycle.  The  next  four  sections  described  the  concepts  of  the  Input,  Processing,  Storage  and  Output 
operations  in  a computer  and  explained  the  function  of  the  different  components  associated  with  that  operation. 
The  content  used  in  this  study  was  part  of  the  required  content  for  the  couise.  The  computer-based  lesson  was 
pilot  tested  with  five  students  before  it  was  used  in  the  study. 

The  material  was  designed  in  six  different  versions  that  included  various  combinations  of  Gagne’s 
instructional  events  as  described  above.  The  six  versions  consisted  of  (1)  a full  version  that  contained 
information  plus  all  events  investigated  in  the  study,  (2)  a version  without  objectives  (3)  one  without  examples, 
(4)  one  without  practice,  (5)  one  without  review  and  (6)  a lean  version  containing  information  only.  The 
systematic  deletion  of  individual  events  permitted  the  study  of  the  program  both  with  and  without  each  event. 
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Procedures 

Eighteen  sections  of  students  (n  = 256)  enrolled  in  the  Computer  Literacy  Course  were  randomly 
assigned  to  the  six  treatment  groups  based  on  pretest  scores.  The  pretest,  which  took  approximately  15  minutes 
to  complete,  was  administered  three  weeks  prior  to  the  study.  The  classes  were  blocked  into  three  groups  (high, 
medium  and  low)  based  on  their  mean  pretest  scores,  and  one  class  within  each  block  was  randomly  assigned  to 
each  of  the  six  treatments. 

The  participants  participated  in  the  web-based  IPSO  lesson  during  the  sixth  week  of  the  semester. 
Participants  met  in  a regular  computer  lab  for  instruction  and  were  directed  by  the  instructor  to  the  web  address 
for  the  instructional  program.  Each  class  was  routed  directly  to  its  treatment  version  of  the  program.  Students 
worked  through  the  program  at  their  own  pace,  averaging  approximately  one  hour.  Then  they  took  the  posttest 
and  the  attitude  survey  online.  All  six  treatment  groups  followed  the  same  procedure.  Thus,  the  experimental 
differences  in  treatments  occurred  exclusively  in  the  materials  themselves  and  not  in  the  procedure. 

Criterion  Measures 

The  criterion  measures  consisted  of  a posttest  and  a student  attitude  survey.  A pretest  was  used  to 
assess  subject’s  knowledge  of  the  content  prior  to  the  instruction  and  to  randomly  assign  classes  within  ability 
blocks  to  treatment. 

Pretest  - The  pretest  consisted  of  20  multiple -choice  questions  covering  the  content  with  four  response 
choice  questions.  The  overall  mean  score  on  the  pretest  was  8.68  or  43%,  indicating  that  participants  were  not 
very  knowledgeable  about  the  content  prior  to  instruction.  Thus  participants  had  relatively  little  knowledge  of 
content  prior  to  instruction.  There  were  no  significant  differences  across  the  six  treatment  groups  in  pretest 
scores. 

Posttest  - The  posttest  consisted  of  the  same  20  multiple -choice  questions  that  were  on  the  pretest.  It 
was  judged  to  be  unlikely  that  the  pretest  would  have  an  effect  on  posttest  scores  that  could  be  a threat  to 
validity  (Campbell  & Stanley,  1963)  because  of  the  three-week  interval  between  test  and  the  fact  that  feedback 
was  not  given  on  the  pretest.  The  posttest  score  was  counted  towards  their  course  grade,  and  this  motivated  the 
learners  to  learn  from  the  web-based  lesson. 

Attitude  Survey  - The  attitude  survey  assessed  student  attitudes  towards  the  instructional  program  and 
the  presence  or  absence  of  the  instructional  elements.  The  12-item  survey  consisted  of  Likert -type  questions  that 
were  rated  strongly  agree  (scored  as  4)  to  strongly  disagree  (scored  as  0).  The  survey  was  administered  after  the 
lesson  and  the  posttest  were  completed. 

Data  Analysis 

A one-way  ANOVA  was  conducted  to  analyze  the  posttest  data  for  statistical  significance.  A 
MANOVA  was  conducted  on  the  12  attitude  questions.  Both  analyses  revealed  significant  differences. 
Therefore,  Scheffe  tests  were  performed  for  both  data  sets  to  test  for  significance  between  groups.  Alpha  was 
set  at  .01  for  all  statistical  tests  because  of  the  large  number  of  comparisons. 

Results 


Achievement 

Table  1 shows  the  mean  scores  and  standard  deviations  by  treatment  for  achievement  on  the  posttest. 
The  table  shows  that  the  mean  scores  for  subjects  in  each  of  the  four  treatments  (full  program,  program  without 
objectives,  program  without  examples,  and  program  without  review)  were  above  17  items  correct,  whereas  the 
scores  for  the  other  two  treatments  (program  without  practice  and  lean  program)  were  below  15  correct.  The 
table  also  shows  that  the  mean  posttest  score  across  all  six  treatments  was  16.44  items  correct. 

Table  1 Means  and  Standard  Deviations  for  Posttest  Scores  by  Treatment 


Treatment 

Posttest 

Posttest 

Mean 

SD 

Full  Program 

17.61 

1.99 

Program  without  Objectives 

17.36 

1.75 
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Program  without  Examples 

17.16 

2.34 

Program  without  Review 

17.17 

2.76 

Program  without  Practice 

14.98 

2.66 

Lean  Program 

14.73 

3.21 

Total 

16.44 

2.75 

A one-way  ANOVA  conducted  on  the  posttest  data  yielded  a significant  difference  between  the 
treatment  groups  on  the  posttest,  F (5,  250)  = 11.689,  p <.01.  Follo\wup  Scheffe  tests  revealed  that  each  of  the 
four  groups  with  means  scores  above  17,  as  listed  above  and  shown  in  the  table,  scored  significantly  higher  than 
the  two  groups  identified  above  that  scored  below  15.  There  were  no  significant  differences  between  the  four 
groups  scoring  above  17  or  between  the  two  scoring  below  15. 

Attitude 

Table  2 shows  means  for  responses  to  the  12  Likert -type  items  on  the  attitude  survey.  The  items  were 
rated  on  a 5-  point  Likert  scale  from  strongly  agree  (scored  as  4)  to  strongly  disagree  (scored  as  0). 


Table  2 Attitude  Scores  by  Treatment 


Attitude 

Questions 

*FP 

NO 

NE 

NR 

NP 

LP 

Total 

1 

The  goals  of  the  program  were  clear  to 
me. 

**3.3 

4 

3.40 

3.22 

3.34 

3.17 

2.77 

3.21 

2 

I knew  what  I was  supposed  to  learn  at 
the  start  of  each  section  of  the  program. 

3.07 

3.16 

3.11 

2.98 

2.96 

2.68 

2.99 

3 

The  program  included  enough  pictures 
and  examples. 

3.20 

3.38 

2.41 

3.10 

3.06 

2.43 

2.93 

4 

The  graphics  helped  me  understand  the 
content  well. 

3.17 

3.32 

2.30 

3.12 

2.83 

2.36 

2.85 

5 

The  review  at  the  end  of  each  section 
helped  my  learning. 

3.63 

3.74 

3.43 

3.39 

3.17 

2.68 

3.34 

6 

The  program  had  enough  opportunity  to 
review  the  content. 

3.17 

3.18 

3.07 

2.76 

2.77 

2.48 

2.91 

7 

The  practice  in  the  program  helped  me 
learn  the  content. 

3.34 

3.52 

3.28 

3.05 

2.15 

2.39 

2.96 

8 

The  program  gave  me  enough 
opportunity  to  practice  what  I was 
learning. 

3.05 

3.20 

3.04 

2.56 

2.13 

2.25 

2.71 

9 

I learned  a lot  from  this  program. 

3.02 

3.22 

2.91 

2.88 

2.63 

2.64 

2.88 

10 

I would  recommend  this  program  to 
other  students. 

3.07 

3.30 

2.72 

2.83 

2.52 

2.45 

2.82 

11 

I would  enjoy  using  other  computer 
programs  like  this  one  in  future  lessons. 

2.85 

3.26 

2.80 

2.56 

2.58 

2.36 

2.74 

12 

The  overall  quality  of  the  program  was 
good. 

3.20 

3.34 

3.11 

2.98 

2.88 

2.70 

3.04 

Average 

3.18 

3.34 

2.95 

2.96 

2.74 

2.52 

2.95 
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* Note.  FP  - Full  Program  NO-  No  Objectives  Group 

NP  - No  Practice  Group  NE  - No  Examples  Group 

NR  - No  Review  Group  LP  - Lean  Program 

**  Note.  4=  Strongly  Agree  3=  Agree  2=  Neither  agree  nor  disagree  1=  Disagree  0 = Strongly  Disagree 

A MANOVA  conducted  on  the  overall  attitude  data  revealed  a significant  overall  difference  on  the  12 
attitude  questions,  F (60,  1188.48)  = 12.98,  p <.01.  Follow-up  univariate  analyses  indicated  significant 
differences  on  1 1 of  the  12  attitude  survey  items  at  the  p<.01  level.  The  only  item  that  did  not  show  a significant 
difference  at  this  level  was  “I  knew  what  I was  supposed  to  learn  at  the  start  of  each  section  of  the  program.” 

The  eleven  items  on  which  significance  was  obtained  were  further  analyzed  to  identify  significant 
differences  between  treatment  groups  on  these  items.  Table  3 provides  a summary  of  the  significant  differences 
found  when  follow-up  Scheffe  tests  were  conducted  at  the  .01  level.  These  data  show  that  participants  who  used 
the  program  without  objectives  had  the  most  positive  attitudes  toward  their  treatment  with  17  significant 
comparisons.  Participants  who  used  the  lean  program  had  the  most  negative  attitudes  toward  their  treatment 
with  21  significant  negative  comparisons.  Those  who  used  the  program  without  practice  had  10  significant 
negative  comparisons. 

Table  3 Summary  of  Significantly  Higher  and  Lower  Differences  for  Student  Attitudes 


Treatments 

*Significantly  Higher 

**Significantly  Lower 

Full  Program 

10 

Program  without  Objectives 

17 

Program  without  Examples 

5 

7 

Program  without  Review 

5 

Program  without  Practice 

1 

10 

Lean  Program 

21 

* Note.  Indicates  the  number  of  between-group  comparisons  of  mean  scores  across  the  12  attitude  items  that 
were  significantly  more  positive  for  each  group. 

**  Note.  Indicates  the  number  of  between-group  comparisons  of  mean  scores  across  the  12  attitude  items  that 
were  significantly  more  negative  for  each  group 

On  the  attitude  items  regarding  practice  --  ‘‘The  practice  in  the  program  helped  me  learn  the  content” 
(Item  7)  and  ‘‘The  program  gave  me  enough  opportunity  to  practice  what  I was  learning”  (Item  8)  --  participants 
in  each  of  the  two  treatments  that  did  not  include  practice  had  significantly  lower  attitudes  than  those  in  each  of 
the  four  treatments  that  included  practice.  On  the  attitude  items  related  to  examples  --  ‘‘The  program  included 
enough  pictures  and  examples”  (Item  3)  and  ‘‘The  graphics  helped  me  understand  the  content  well”  (Item  4)  -- 
participants  in  the  two  treatments  that  did  not  include  examples  had  the  lowest  attitudes  compared  to  those  in 
the  treatments  that  provided  examples.  However,  on  the  attitude  items  regarding  objectives  --  ‘‘The  goals  of  the 
program  were  clear  to  me”  (Item  1),  and  “I  knew  what  I was  supposed  to  learn  at  the  start  of  each  section  of  the 
program”  (Item  2)  — participants  in  the  program  without  objectives  gave  the  highest  overall  ratings  of  all  six 
groups,  though  not  significantly  higher  than  most  groups. 

The  attitude  survey  also  included  two  open-ended  questions  that  asked  the  participants  what  they  liked 
best  and  least  about  the  program.  The  most  frequent  responses  for  what  participants  liked  best  were  the  review 
section  (n= 63),  the  practice  questions  («=  59),  examples/graphics  («=37),  and  easy  to  use/usability  («= 33).  The 
most  frequent  responses  for  what  was  liked  least  were  lots  of  information  (/)=  54),  length  of  the  program 
(«= 39),  and  inability  to  go  back  to  the  previous  screen  (h=10).  Twenty-four  participants  mentioned  that  there 
was  nothing  they  disliked  about  the  program. 


Discussion 

This  study  examined  the  effects  of  instructional  events  (information,  objectives,  examples,  practice  and 
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review)  on  achievement  and  attitudes.  College  students  enrolled  in  a computer  literacy  course  used  a computer- 
based  lesson  delivered  on  the  web  to  learn  about  input,  processing,  storage  and  output  of  a computer  (IPSO). 

Results  indicated  that  among  the  instructional  events,  practice  had  the  most  impact  on  both  learner 
achievement  and  attitudes.  Participants  who  used  one  of  the  versions  of  the  computer  program  that  included 
practice  (full  program,  program  without  objectives,  program  without  examples  and  program  without  review) 
performed  significantly  better  on  the  posttest  than  those  who  did  not  receive  practice  (program  without  practice 
and  lean  program).  Furthermore,  students  who  received  practice  in  their  program  had  consistently  more  positive 
attitudes  than  those  who  did  not  receive  it. 

Achievement 

Practice  was  clearly  the  instructional  event  that  had  the  strongest  positive  effect  on  achievement  among 
the  events  manipulated  in  this  study.  Participants  in  all  four  treatments  that  included  practice  scored 
significantly  higher  on  the  posttest  than  those  in  the  two  conditions  that  did  not  include  it.  Whereas  the  removal 
of  practice  from  the  full  program  resulted  in  a significant  decline  in  posttest  performance,  removal  of  any  one  of 
the  three  other  events  (objectives,  examples,  review)  did  not  have  such  an  effect. 

Practice  of  the  type  in  this  study  is  effective  because  it  gives  learners  the  opportunity  to  perform  a 
similar  or  identical  learning  task  to  that  assessed  on  the  posttest.  This  practice  combined  with  feedback,  as  it 
was  in  the  present  study,  enables  learners  to  confirm  their  correct  understanding  and  to  identify  their  incorrect 
ones.  Thus,  the  probability  of  retention  of  correct  responses  is  increased  and  the  probability  of  incorrect 
responses  is  reduced  when  the  practice  is  aligned  with  the  subsequent  posttest  assessment  (Philips  et  ah,  1988; 
Reiser  & Dick,  1996).  Practice  also  has  the  advantage  of  eliciting  overt  responses  from  the  learner,  a form  of 
active  participation  not  directly  provided  by  the  other  elements  of  instruction  investigated  in  this  study. 

Whereas  practice  elicits  overt  responding  from  learners,  the  other  elements  investigated  in  the  study 
either  provide  information  that  is  additional  to  that  contained  in  the  information  screens  (that  is,  the  objectives) 
or  that  is  supplementary  (examples)  or  primarily  redundant  (review)  to  the  information.  The  absence  of  each  of 
these  elements  individually  in  one  of  the  three  different  treatments  in  the  present  study  (program  without 
objectives,  without  examples  or  without  review)  consistently  yielded  a posttest  score  between  17.16  and  17.36 
that  varied  only  slightly  and  non -significantly  from  the  score  of  17.61  for  students  in  the  full  program.  Thus, 
there  is  no  evidence  from  this  study  that  any  of  these  three  elements  individually  contributed  to  increased 
student  learning.  Hannafin  (1987)  noted  that  when  computer-based  instruction  is  systematically  designed,  the 
presence  of  objectives  for  students  may  not  increase  their  achievement.  Nevertheless,  the  presence  of  objectives 
may  be  essential  for  the  instructional  designer  to  design  the  instruction  systematically. 

Attitudes 

Turning  to  attitudes,  results  revealed  that  most  participants  had  a favorable  impression  about  the 
computer-based  lesson  used  in  this  study.  In  general,  they  agreed  with  statements  such  as,  “I  learned  a lot  from 
this  program,”  “I  would  recommend  this  program  to  other  students,”  and  ‘‘The  overall  quality  of  the  program 
was  good.” 

Results  for  attitudes  were  generally  consistent  with  findings  for  achievement.  When  the  items  on  the 
attitude  survey  were  analyzed  to  examine  differences  between  treatment  groups,  participants  who  used  the  lean 
program  had  the  most  negative  attitudes  toward  their  treatment  followed  by  those  who  used  the  program 
without  practice.  Combined  with  results  for  achievement,  this  study  suggests  that  practice  not  only  increases 
learning,  but  the  absence  of  it  also  diminishes  students’  attitudes  toward  instruction. 

Student  responses  to  the  attitude  survey  showed  that  they  were  sensitive  to  the  absence  of  some  of  the 
instructional  elements  investigated  in  this  study.  Participants  who  received  practice  in  their  program  agreed 
significantly  more  with  items  related  to  the  amount  and  helpfulness  of  the  practice  than  students  who  did  not 
receive  practice.  Furthermore,  participants  who  received  examples  throughout  the  program  agreed  more  with 
items  related  to  the  amount  and  usefulness  of  the  examples  than  students  who  did  not  receive  exa  mples.  These 
findings  suggest  that  students  are  aware  when  practice  and  examples  are  left  out  of  computer-based  instruction 
and  that  excluding  these  elements  has  a detrimental  effect  on  their  attitudes. 

However,  this  pattern  was  not  found  for  the  attitude  items  related  to  objectives.  Students  in  the  no- 
objectives treatment  had  the  most  positive  responses  to  the  two  items  related  to  the  goals  and  objectives  of  the 
program.  They  also  had  significantly  more  positive  attitudes  toward  their  treatment  when  their  results  were 
compared  with  students  in  several  of  the  other  treatments.  This  finding  suggests  that  students  may  be  unaware 
of  the  absence  of  objectives  when  other  elements  such  as  practice  are  included  in  the  program. 

In  addition,  students  may  not  always  be  aware  of  the  absence  of  review  in  computer-based  instruction. 
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Participants  in  the  no-review  treatment  and  those  in  the  lean  treatment  did  not  receive  review  throughout  their 
program.  Nevertheless,  students  in  the  no -review  condition  had  significantly  rnoie  positive  response  than  those 
in  the  lean  group  on  the  item,  “The  review  at  the  end  of  each  section  helped  my  learning.”  It  should  be  noted 
that  when  asked  what  they  liked  best  about  the  program,  students  most  frequently  listed  the  element  of  review. 

Implications  and  Future  Research 

This  study  has  implications  for  the  design  and  development  of  computer-based  instruction.  Practice 
was  the  one  consistently  effective  instructional  event  for  enhancing  student  achievement  in  the  study.  This 
suggests  that  it  should  be  included  in  computer-based  instruction  especially  when  students  are  tested  using 
items  aligned  with  the  objectives  and  practice  items.  However,  the  lack  of  effect  produced  by  the  one-at-a  time 
removal  of  objectives,  examples  and  review  from  individual  treatment  versions  should  not,  of  course,  be 
interpreted  as  an  indicator  that  one  or  more  of  these  elements  should  routinely  be  deleted  from  an  instructional 
program.  If  a program  is  well  conceptualized,  none  of  the  three  is  very  costly  in  terms  of  writing  time  by  the 
designer,  amount  of  text  space  in  the  lesson,  or  length  of  reading  time  by  the  learner.  These  three  elements  were 
included  and  removed  systematically  in  the  present  study  in  order  to  investigate  their  effects  in  a controlled 
instructional  environment.  Their  desirability,  and  possibly  their  effects,  may  vary  in  other  settings  depending  on 
such  factors  as  the  age  and  motivation  of  the  learners  and  the  complexity  of  the  subject  matter. 

Future  research  should  continue  to  focus  on  the  impact  of  instructional  events  in  various  instructional 
settings.  Additional  research  should  examine  how  instructional  events  in  computer-based  instruction  influence 
outcomes  such  as  problem  solving  and  complex  learning  tasks.  Furthermore,  the  recent  proliferation  of  web- 
based  and  Internet-based  instruction  suggests  that  studies  should  be  conducted  to  examine  the  effect  of 
objectives,  examples,  practice  and  review  in  these  settings.  As  was  done  in  this  study,  research  in  these  settings 
should  include  measures  of  student  achievement  and  attitudes.  Studies  of  this  nature  will  continue  to  inform 
designers  about  the  influence  of  instructional  events  on  learning  and  performance. 
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Introduction 

Recent  instructional  theories  tend  to  focus  on  authentic  learning  tasks  that  are  based  on  real-life  tasks 
as  the  driving  force  for  complex  learning  (Merrill,  2002;  van  Merrienboer  & Kirschner,  2001).  The  general 
assumption  is  that  such  tasks  help  learners  to  integrate  the  knowledge,  skills  and  attitudes  necessary  for 
effective  task  performance;  give  them  the  opportunity  to  learn  to  coordinate  constituent  skills  that  make  up  this 
performance,  and  eventually  enable  them  to  transfer  what  is  learned  to  their  daily  life  or  work  settings.  This 
focus  on  authentic,  whole  tasks  can  be  found  in  several  educational  approaches,  such  as  the  case  method, 
project-based  education,  problem-based  learning,  and  competency-based  education.  Van  Merrienboer’ s four- 
component  instructional  design  model  (4C/ID-model,  1997;  van  Merrienboer,  Clark,  & de  Croock,  2002) 
describes  how  learning  tasks  fulfill  the  role  of  a backbone  for  an  integrated  curriculum  (Figure  1:  the  “circles” 
represent  learning  tasks).  Two  requirements  for  this  backbone  are:  (a)  learning  tasks  are  organized  in  easy-to- 
difficult  task  classes  (the  dotted  boxes  around  sets  of  learning  tasks),  and  (2)  learners  receive  guidance  for  the 
first  learning  task  in  a task  class  after  which  support  slowly  disappears  in  this  task  class. 


Figure  1.  A sequence  of  learning  tasks,  organized  in  easy-to-difficult  task  classes  and  with  fading  support  in  each  task  class 


It  is  clearly  impossible  to  use  very  difficult  learning  tasks  right  from  the  start  of  a curriculum  or 
educational  program  because  this  would  yield  excessive  cognitive  load  for  the  learners,  with  negative  effects  on 
learning,  performance,  and  motivation  (Sweller,  van  Merrienboer,  & Paas,  1998;  van  Merrienboer,  Kirschner,  & 
Kester,  2003).  The  common  solution  is  to  let  learners  start  their  work  on  relatively  easy  learning  tasks  and 
progress  towards  more  difficult  tasks.  In  a whole -task  approach,  the  coordination  and  integration  of  constituent 
skills  is  yet  stressed  from  the  very  beginning,  so  that  learners  quickly  develop  a holistic  vision  of  the  whole  task 
that  is  gradually  embellished  during  the  training.  This  is  akin  to  the  “global  before  local  skills”  principle  in 
cognitive  apprenticeship  (Collins,  Brown,  & Newman,  1989,  p.  485)  or  the  “zoom  lens  metaphor”  of 
Reigeluth's  elaboration  theory  (1999).  There  are  categories  of  learning  tasks  or  task  classes,  each  representing  a 
version  of  the  task  with  a particular  difficulty  (the  dotted  boxes  in  Figure  1).  Learning  tasks  within  a particular 
task  class  are  equivalent  in  the  sense  that  the  tasks  can  be  performed  on  the  basis  of  the  same  body  of 
generalized  knowledge.  A more  difficult  task  class  requires  more  knowledge  or  more  embellished  knowledge 
for  effective  performance  than  the  preceding,  easier  task  classes.  In  other  words,  each  new  task  class  contains 
learning  tasks  that  are  in  the  zone  of  proximal  development  of  the  learners  (Vygotsky,  1934/1987).  It  is 
essential  that  the  equivalent  learning  tasks  within  the  same  task  class  show  a high  variability,  that  is,  differ  from 
each  other  in  terms  of  the  saliency  of  defining  characteristics,  the  context  in  which  the  task  has  to  be  performed, 
the  familiarity  of  the  task,  or  any  other  task  dimensions  that  also  vary  in  the  real  world  (Paas  & van 
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Merrienboer,  1994).  Variability  is  a key  factor  for  reaching  the  necessary  level  of  generality  and  facilitating 
transfer  of  learning  to  daily  life  or  future  work  settings. 

Furthermore,  when  learners  start  to  work  on  a new,  more  difficult  task  class,  it  is  essential  to  give  them 
guidance  and  support.  This  support  diminishes  in  a process  of  “scaffolding”  as  learners  acquire  more  expertise 
(see  the  filling  of  the  circles  in  Figure  1).  One  powerful  approach  to  scaffolding  is  known  as  the  “completion 
strategy”.  In  this  strategy,  learners  start  to  work  on  fully  worked-out  examples  that  confront  them  not  only  with 
a given  problem  state  and  a desired  goal  state,  but  also  with  an  example  solution.  Questions  and  evaluation 
assignments  stimulate  the  learners  to  reflect  on  the  strong  and  weak  points  of  the  given  solution.  Studying 
worked-out  examples  focuses  the  learners’  attention  on  problem  states  and  associated  solution  steps  and  so 
enables  them  to  induce  generalized  solutions  or  schemata.  Then,  learners  may  proceed  to  work  on  completion 
tasks  that  present  a given  state,  a goal  state,  and  a partial  solution  that  must  be  completed.  There  is  still  a 
sizeable  support,  because  the  given  part  of  the  solution  provides  direction  to  the  problem  solving  process. 
Finally,  learners  receive  conventional  tasks  without  support  - only  then,  they  have  to  construct  complete 
solutions.  Several  studies  showed  positive  effects  on  learning  for  the  completion  strategy  (Renkl  & Atkinson, 
2003;  van  Merrienboer  & de  Croock,  1992). 

In  a flexible  curriculum,  it  should  be  possible  to  take  differences  between  students  into  account.  Some 
students  are  better  able  to  acquire  new  complex  skills  or  competencies  and  need  therefore  less  practice  and 
support  than  other  students.  In  addition,  elsewhere -acquired  skills  of  new  students  should  be  taken  into  account. 
And  complex  skills  or  competencies  are  not  coupled  to  separate  courses  or  modules  but  developed  throughout 
the  curriculum  or  educational  program,  which  makes  it  even  more  important  to  be  able  to  select  suitable 
learning  tasks  for  students.  In  the  4C/ID- framework  sketched  above,  this  means  that  for  each  individual  student, 
it  should  be  possible  to  select  the  best  task  class  to  work  on,  and  to  select  within  this  task  class  a learning  task 
with  the  optimal  level  of  support,  at  any  given  point  in  time.  Electronic  learning  environments  allow  for  such 
dynamic  selection  of  learning  tasks. 

Dynamic  Task  Selection  on  the  Basis  of  Mental  Efficiency 

Models  for  dynamic  task  selection  typically  take  learner’s  performance  as  their  input,  defined  in  terms 
of  the  number  of  correctly  answered  test  items,  the  number  of  errors,  or  the  time  on  task.  However,  the  4C/ID- 
rnodel  stresses  that  other  dimensions  are  at  least  equally  important  for  the  assessment  of  expertise.  They  include 
mental  load,  which  originates  from  the  interaction  between  task  characteristics  (e.g.,  task  format,  multimedia, 
task  difficulty)  and  learner  characteristics  (e.g.,  age,  prior  knowledge,  spatial  ability)  and  so  yields  an  a-priori 
estimate  of  cognitive  load,  and  mental  effort,  which  refers  to  the  cognitive  capacity  that  is  actually  allocated  to 
accommodate  the  demands  imposed  by  the  task  (Paas  & van  Merrienboer,  1993,  1994b).  Especially  mental 
effort  may  yield  important  information  that  is  not  necessarily  reflected  in  mental  load  and  performance 
measures.  For  instance,  it  is  quite  feasible  for  two  persons  to  attain  the  same  performance  levels  with  one  person 
needing  to  work  laboriously  through  a very  effortful  process  to  arrive  at  the  correct  answers,  whereas  the  other 
person  reaches  the  same  answers  with  a minimum  of  effort.  While  both  people  demonstrate  identical 
performance,  ‘expertise’  may  be  argued  to  be  higher  for  the  person  who  performs  the  task  with  minimum  effort 
than  for  the  person  who  exerts  substantial  effort. 

An  appropriate  assessment  of  expertise  should  thus  at  least  include  measures  of  mental  effort  and 
performance.  Paas,  Tuovinen,  Tabbers  and  van  Gerven  (2003)  discuss  different  measurement  techniques  for 
mental  effort,  including  rating  scales,  secondary  task  methods,  and  psychophysiological  measures.  On  the  basis 
of  a comprehensive  review  of  about  30  studies,  they  conclude  that  “...the  use  of  rating  scales  to  measure  mental 
effort  remains  popular,  because  they  are  easy  to  use;  do  not  interfere  with  primary  task  performance;  are 
inexpensive;  can  detect  small  variations  in  workload  (i.e.,  sensitivity);  are  reliable,  and  provide  decent 
convergent,  construct,  and  discriminate  validity”  (p.  68).  For  the  measurement  of  complex  performance,  several 
methods  that  assess  and  weigh  different  aspects  of  performance  have  been  developed  (see  Hambleton,  Jaeger, 
Plake,  & Mills,  2000).  However,  most  methods  are  very  time-consuming.  To  make  the  assessment  of  complex 
performance  more  time  -effective,  Kalyuga  and  Sweller  (in  press)  proposed  a “rapid  assessment  test”,  which 
asked  students  to  indicate  their  first  step  toward  solution  of  a complex  task.  High  correlations  (up  to  .92)  were 
found  between  performance  on  “rapid  assessment  tests”  and  traditional  performance  tests  that  required 
complete  solutions  of  corresponding  tasks. 

A final  step  in  the  assessment  of  expertise  is  the  difficult  task  of  combining  a student’s  mental  effort 
and  performance  measures,  because  a meaningful  interpretation  of  a certain  level  of  invested  effort  can  only  be 
given  in  the  context  of  its  associated  performance  and  vice  versa.  Paas  and  van  Merrienboer  (1993;  see  also 
Paas  et  ah,  2003)  developed  a computational  approach  to  combine  measures  of  mental  effort  with  measures  of 
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associated  performance  to  compare  the  mental  efficiency  associated  with  instructional  conditions  - under  the 
assumption  that  learners’  behavior  in  a particular  condition  is  more  efficient  if  their  performance  is  higher  than 
might  be  expected  on  the  basis  of  their  invested  mental  effort  or,  equivalently,  if  their  invested  mental  effort  is 
lower  than  might  be  expected  on  the  basis  of  their  performance.  Using  this  approach,  high  task  performance 
associated  with  low  effort  is  called  high  mental  efficiency,  whereas  low  task  performance  with  high  effort  is 
called  low  mental  efficiency.  Unfortunately,  this  approach  can  only  be  used  after  all  data  of  a group  of  students, 
working  in  different  instructional  conditions,  have  been  gathered.  Alternative  methods  are  needed  for  the 
continuous  assessment  of  expertise  of  individual  learners.  Such  alternatives  are  currently  under  development  in 
the  context  of  adaptive  eLearning. 


Adaptive  E-Learning 

Salden,  Paas  and  van  Merrienboer  (in  press)  discuss  the  value  of  the  4C/ID -model  for  adaptive 
eLearning,  with  a focus  on  the  dynamic  selection  of  learning  tasks.  They  describe  adaptive  eLearning  as  a 
straightforward  two-step  cycle:  (1)  assessment  of  a learner’s  expertise,  and  (2)  task  selection.  With  regard  to  the 
ongoing  assessment  of  expertise,  they  differentiated  between  a learner  who  needs  to  work  laboriously  to  attain  a 
certain  performance  level  (low  mental  efficiency)  and  a learner  who  attains  the  same  performance  level  with 
little  mental  effort  (high  mental  efficiency).  Only  the  second  learner  who  solved  the  problem  efficiently  should 
be  presented  with  a more  difficult  and/or  less-supported  learning  task.  With  regard  to  task  selection,  given  the 
learner’s  mental  efficiency  one  might  select  tasks  (1)  that  provide  less,  equal,  or  more  support  to  learners  than 
the  previous  task(s);  (2)  that  are  less,  equally,  or  more  difficult  than  the  previous  task(s),  and  (3)  that  vary  with 
regard  to  both  support  and  difficulty. 

Selecting  Learning  Tasks  with  Different  Levels  of  Support 

In  a study  reported  by  van  Merrienboer,  Schuurman,  de  Croock,  and  Paas  (2002),  participants  received 
a 3-hour  introductory  computer-programming  course  in  the  computer-based  learning  environment  CASCO 
(Completion  Assignment  Constructor;  van  Merrienboer  & Luursema,  1996).  Participants  received  no  support 
(i.e.,  conventional  programming  tasks;  n = 8),  support  (i.e.,  completion  tasks;  n = 10),  or  adaptive  support  (n  = 
8).  In  the  no-support  and  support  conditions,  each  new  learning  task  was  selected  from  a database  of  tasks  in 
such  a way  that  the  selected  task  offered  the  best  opportunity  to  practice  those  programming  concepts  that  were 
not  yet  mastered  by  the  student.  In  the  adaptive  support  condition,  students  selected  completion  tasks  based  on  a 
subjective  estimate  of  their  mental  efficiency:  The  tasks  could  range  from  fully  worked-out  examples  to 
conventional  tasks,  that  is,  in  their  level  of  build-in  support.  All  tasks  that  could  be  presented  to  the  learners 
were  of  roughly  the  same  difficulty  level,  and  thus  only  differed  with  regard  to  the  programming  concepts  that 
had  to  be  practiced  and,  for  the  adaptive  condition,  the  amount  of  given  support.  Learning  tasks  consisted  of  a 
problem  statement  and  (1)  explanations,  concerning  new  programming  concepts  that  were  necessary  for  writing 
the  program,  (2)  specific  subtasks  that  could  help  to  write  the  program,  and  (3)  questions  that  were  relevant  for 
the  task  at  hand.  For  completion  tasks,  a partial,  to-be-completed  program  was  presented  in  a full-fledged  editor 
window;  for  conventional  tasks,  this  editor  window  was  empty.  The  CASCO  interface  is  presented  in  Figure  2. 


File 


CASCO  - Piet  - Seconds  after  midnight 


Assignment  Back(F5)  Continue(F6) 


Debug  Help 


Screen  layout 


1 PROGRAM 

2 Specification  : Seconds  after  midnight 

3 explanation  about  Using  named  constants 
J 4 task  about  Prompting  the  user  for  input 

5 task  about  Print  explanatory  text 


100:  PRINT 

300:  SecPerMin  :=  60 
400:  MinPei'Hou?  :=  60 
500:  Secs  :=  TotalSec  MOD  SecPerMin 
600:  Mins  :=  (TotalSec  DIU  SecPerMin)  MOD  MinPerHour 
700:  Hours  :=  TotalSec  DIU  (SecPerMin  * MinPerHouiO 
800:  END 


Specification  . Seconds  after  midnight 


Write  a program  that  prints  a 
number  of  seconds  in  the  format 
'hours:minutes:secondsa. 


task  about  Prompting  the  user  for  input 


The  program  asks  the  user  to  input  a number,  in  line  200. 
Complete  this  line  with  the  name  of  the  right  variable. 


Help 


I don't  know 


TTT7- 


Figure  2.  The  CASCO  Interface. 


Practice  data  show  that  learners  in  the  support  group  finished  the  highest  number  of  learning  tasks  in 
the  three-hour  practice  phase  (M  = 28. 1),  compared  to  the  no-support  group  (M  = 8.3)  and  the  adaptive  support 
group  (M  = 21.3;  F(2,  26)  = 13.7,  MSE=  66.4,  p < .001).  In  post-hoc  tests,  using  Tukey’s  HSD,  it  was  found 
that  both  the  no -support  and  the  support  group  (p  < .001)  and  the  no-support  and  the  adaptive  support  group  (p 
< .01)  differed  significantly.  For  a transfer  test  that  was  performed  after  the  training,  the  proportion  of  correctly 
used  programming  concepts  was  .33  for  the  no-support  group,  .39  for  the  support  group,  and  .55  for  the 
adaptive  support  group.  ANOVA  indicated  a significant  difference  between  conditions,  F(2  ,26)  = 3.6,  MSE  = 
.03,  p < .05.  As  predicted,  the  adaptive  support  group  outperformed  the  support  and  no-support  groups. 
Concluding,  adapting  the  level  of  support  to  the  learners  had  beneficial  effects  on  learning  and  transfer  test 
performance. 


Selecting  Learning  Tasks  with  Different  Levels  of  Difficulty 

In  the  domain  of  Air  Traffic  Control  (ATC),  Camp,  Paas,  Rikers  and  van  Merrienboer  (2001)  and 
Salden,  Paas,  Broers  and  van  Merrienboer  (2004)  compared  the  effectiveness  of  a fixed  easy-to-difficult 
sequence  of  learning  tasks  with  dynamic  task  selection  based  on  mental  efficiency.  In  the  mental  efficiency 
condition,  learners  received  ATC  tasks  at  possible  10  levels  of  difficulty,  starting  at  the  lowest  level.  Depending 
on  the  assessment  results,  the  next  task  was  selected.  For  instance,  a student  who  attains  a performance  score  of 
4 while  his  or  her  mental  effort  is  3 will  be  presented  with  a learning  task  that  is  one  difficulty  level  higher  than 
the  previous  task;  another  student  who  attains  a performance  score  of  4 while  his  or  her  mental  effort  is  only  1 
will  be  presented  with  a learning  task  that  is  two  difficulty  levels  higher  than  the  previous  task.  In  both  studies, 
dynamic  task  selection  yielded  more  efficient  transfer  test  performance  than  the  use  of  a fixed  sequence  of 
tasks.  The  mental  efficiency  condition  was  also  more  effective  during  training  than  the  fixed  condition: 
Participants  needed  fewer  learning  tasks  to  reach  the  highest  difficulty  level,  reached  a higher  difficulty  level, 
and  made  larger  jumps  to  higher  difficulty  levels  than  students  in  the  fixed  condition. 

In  a just  completed  study,  participants  learned  to  use  a Flight  Management  System  (FMS)  according  to 
either  (a)  a fixed  easy-to-difficult  sequence  of  16  learning  tasks  (n  = 10),  (b)  a system-controlled  mental 
efficiency  condition  (n  = 1 1),  and  (c)  a learner-controlled  mental  efficiency  condition  (n  = 10).  Prior  to  training, 
the  thirty-two  learning  tasks  were  categorized  into  eight  difficulty  levels  (four  tasks  per  level;  note  that  only  two 
of  those  four  tasks  were  used  in  the  fixed  condition).  In  the  system-controlled  mental  efficiency  condition, 
performance  and  mental  effort  were  measured  and  used  to  determine  the  difficulty  of  the  next  learning  task 
according  to  a table  that  specified  the  increase/decrease  in  difficulty  for  each  combination  of  mental  effort  and 
performance.  For  instance,  if  a participant  had  a mental  effort  score  of  2 and  a performance  score  of  5 (both 
measured  on  a 5-point  scale),  task  difficulty  was  increased  with  three  levels  (+3);  if  a participant  had  a mental 
effort  score  of  5 and  a performance  score  of  3,  task  difficulty  was  decreased  with  two  levels  (-2),  and  so  forth. 

In  the  learner-controlled  mental  efficiency  condition,  participants  were  free  to  select  the  next  task  based  on  a 
subjective  estimate  of  their  mental  efficiency.  The  tasks  were  performed  in  a realistic  computer  simulation  of  a 
Boeing  747  FMS  developed  by  the  National  Aerospace  Laboratory  NLR  (see  Figure  3).  Each  task  presented 
flight  information  of  a certain  route  from  airport  A to  airport  B that  learners  had  to  program  into  the  FMS 
simulation.  A simulated  flight  had  to  be  executed  after  entering  all  information.  At  certain  points  during  the 
task,  changes  in  the  flight  route  were  required  and  made  it  necessary  for  the  trainees  to  adjust  the  original  flight 
route. 
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Figure  3.  The  simulation  of  a Flight  Management  System  (FMS). 


Practice  data  show  that  learners  in  the  mental  efficiency  conditions  needed  substantially  less  than  the 
16  learning  tasks  in  the  fixed  condition  to  complete  the  training:  M = 7.27,  SD  = 1.19  for  the  system-controlled 
condition  (t(20)  = -4.6,  p < .001)  and  M = 6.50,  SD  = 1.35  for  the  learner-controlled  condition  (t(19)  = -4.3,  p < 
.001).  In  line  with  this  finding,  the  mental  efficiency  conditions  also  needed  less  training  time  to  reach  the 
highest  difficulty  level  than  the  fixed  condition  (F(2,  28)  = 28.37,  MSE  = 444.40,  p < .001,  ?2  = .67.  M = 

149.60,  SD  = 22.77  for  the  fixed  condition,  M = 1 17.35,  SD  = 25.61  for  the  system-controlled  mental  efficiency 
condition,  and  M = 78.69,  SD  = 1 1.64  for  the  learner-controlled  condition).  For  a test  with  five  transfer  tasks 
after  the  training,  an  ANCOVA  with  number  of  learning  tasks  and  time -on-task  as  covariates  indicates  that 
participants  scored  2.89  (SD  = .38)  in  the  fixed  condition;  3.21  (SD  = .20)  in  the  system-controlled  mental 
efficiency  condition,  and  3.16  (SD  = .22)  in  the  learner-controlled  mental  efficiency  condition  (Ms  are 
estimated  marginal  means).  Whereas  visual  inspection  indicates  higher  scores  for  the  mental  efficiency 
conditions,  this  difference  does  not  reach  statistical  significance.  However,  the  data  clearly  indicate  that  the 
mental  efficiency  conditions  yield  at  least  the  same  test  performance  with  less  practice  tasks  and  in  less  time 
than  a traditional  fixed  condition. 


Selecting  Learning  Tasks  with  Different  Levels  of  Support  and  Difficulty 

Kalyuga  and  Sweller  (in  press)  conducted  a study  in  which  both  the  difficulty  and  the  given  support  of 
the  next  task  were  adapted  to  the  mental  efficiency  of  the  learner.  They  took  a somewhat  different  approach  to 
combining  performance  and  mental  effort  measures  than  the  previous  studies.  In  the  domain  of  algebra,  a ‘rapid 
assessment  test’  was  used  to  measure  performance  and  a 9-point  rating  scale  was  used  to  measure  mental  effort. 
Cognitive  efficiency  (E)  was  defined  as  a combined  measure  for  monitoring  learners’  progress  during 
instruction  and  real-time  adaptation  of  instructional  formats  to  changing  levels  of  expertise.  Cognitive 
efficiency  is  simply  defined  as  E = P/R,  where  R is  the  mental  effort  rating  and  P is  the  performance  measure  on 
the  same  task.  This  indicator  has  similar  general  features  as  efficiency  defined  by  Paas  and  van  Merrienboer 
(1993),  in  that  it  is  higher  if  similar  levels  of  performance  are  reached  with  less  effort  or,  alternatively,  higher 
levels  of  performance  are  reached  with  the  same  mental  effort.  Students  were  presented  tasks  at  different  levels 
of  difficulty,  and  for  each  level  a critical  level  of  cognitive  efficiency  (Ecr)  was  arbitrarily  defined  as  the 
maximum  performance  score  (which  was  different  per  task  level)  divided  by  the  maximum  mental  effort  score 
(which  was  always  9).  It  should  be  noted  that  this  technique  makes  it  unnecessary  to  use  a baseline  group 
(previous  studies  used  the  “fixed”  group  to  set  this  baseline).  Cognitive  efficiency  is  positive  if  E > Ecr  and 
negative  if  E < Ecr.  The  rationale  for  this  definition  is  that  if  someone  invests  maximum  mental  effort  in  a task 
but  does  not  display  the  maximum  level  of  performance,  his  or  her  expertise  should  be  regarded  as  suboptimal. 
On  the  other  hand,  if  someone  performs  at  the  maximum  level  with  less  than  a maximum  mental  effort,  his  or 
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her  expertise  should  be  regarded  as  optimal. 


STAGE  1 


Figure  4.  Selection  algorithm  governing  the  selection  of  learning  tasks  with  different  levels  of  difficulty  (stages  1-4)  and 
support  (worked-out  examples,  completion  tasks,  and  conventional  tasks/problem  solving  exercises).  Adapted  from  Kalyuga 
and  Swell er  ( in  press) 


In  the  adaptive  group,  learners  were  presented  algebra  tasks  at  three  different  difficulty  levels.  If  their 
cognitive  efficiency  was  negative  for  tasks  at  the  lowest  level,  they  continued  with  the  study  of  worked 
examples;  if  their  cognitive  efficiency  was  positive  for  tasks  at  the  lowest  level  but  negative  for  tasks  at  the 
second  level,  they  continued  with  simple  completion  tasks;  if  their  cognitive  efficiency  was  positive  for  tasks  at 
the  lowest  and  second  level  but  negative  for  tasks  at  the  third  level,  they  continued  with  difficult  completion 
tasks,  and,  finally,  if  their  cognitive  efficiency  was  positive  for  tasks  at  all  three  levels,  they  continued  with 
conventional  problems.  Similar  adaptive  methods  were  applied  when  students  were  working  on  the  worked 
examples,  completion  tasks,  or  conventional  problems  (see  Figure  4).  Each  student  in  the  adaptive  condition 
was  paired  to  a student  in  the  control  condition,  who  served  as  a yoked  control.  Kalyuga  and  Sweller  (in  press) 
report  higher  gains  in  algebraic  skills  from  pretest  to  posttest  and  higher  gains  in  cognitive  efficiency  for  the 
adaptive  eLearning  group  than  for  the  control  group.  Thus,  in  agreement  with  the  other  reported  studies, 
adaptive  eLearning  was  found  to  be  superior  to  non-adaptive  learning. 

Discussion  and  Conclusion 

Modern  instructional  models  stress  the  use  of  whole,  meaningful  learning  tasks  as  the  driving  force  for 
complex  learning.  However,  learners  are  easily  overwhelmed  by  the  complexity  of  such  tasks.  It  is  thus  critical 
to  select  each  new  learning  task  in  such  a way  that  it  is  best  adapted  to  the  individual  needs  of  the  learner. 
According  to  the  4C/ID  model,  adaptation  mainly  refers  to  (a)  the — build-in — support  that  is  provided  to 
learners  while  they  perform  the  task,  and  (b)  the  difficulty  of  the  task  (i.e.,  the  task  class  it  belongs  to).  As  an 
additional  constraint,  the  whole  set  of  tasks  that  is  provided  to  the  learner  must  display  a high  level  of 
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variability.  In  order  to  determine  the  optimal  level  of  support  and  difficulty  of  a next  learning  task,  an  indication 
of  the  learner’s  performance  is  insufficient.  After  all,  if  two  learners  reach  the  same  performance  but  one  needs 
to  put  in  a lot  of  effort  (low  mental  efficiency)  and  the  other  does  it  almost  effortlessly  (high  mental  efficiency), 
the  selected  new  task  should  be  easier  and/or  contain  more  support  for  the  first  learner  than  for  the  second 
learner. 

We  discussed  several  empirical  studies,  which  indicate  that  adapting  the  level  of  support,  the  difficulty 
of  the  learning  task,  or  support  as  well  as  difficulty  has  beneficial  effects  on  learning  and  transfer  test 
performance.  Thus,  adaptive  instruction  is  more  effective  than  non-adaptive  instruction.  Three  conclusions  can 
be  drawn  from  the  presented  studies.  First,  both  the  level  of  available  support  and  the  difficulty  of  the  learning 
task  are  important  dimensions  to  take  into  account  for  dynamic  task  selection.  Second,  performance  alone  is 
often  an  insufficient  basis  for  task  selection  and  should  be  complemented  with,  for  instance,  invested  mental 
effort  so  that  mental  efficiency  drives  the  task  selection.  Third,  the  4C/ID-  model  proved  to  be  not  only  useful  to 
develop  fixed  training  programs  but  also  to  develop  adaptive  forms  of  instruction. 

While  the  presented  results  are  promising,  they  also  yield  important  questions  for  future  research.  A 
first  question  pertains  to  the  complex  relationship  between  difficulty  and  support.  If  a learner’s  mental 
efficiency  is  suboptimal,  should  we  present  a next  task  that  is  equally  difficult  as  the  previous  one  but  with  more 
support;  should  we  present  a next  task  that  is  less  difficult  than  the  previous  one  but  with  the  same  level  of 
support;  or  should  we  vary  both  the  available  level  of  support  and  its  difficulty?  The  4C/ID-model  suggests  to 
vary  only  the  support  until  the  learner  can  perform  the  task  without  any  support,  according  to  relevant  standards 
and  criteria,  and  not  until  then  to  continue  with  more  difficult  learning  tasks  (i.e.,  progress  to  a next  task  class). 
However,  strong  empirical  support  for  this  claim  is  yet  missing.  Second,  it  is  important  to  replicate  our  findings 
on  dynamic  task  selection  with  lengthier  tasks  in  real- life  environments.  Especially  the  measures  of  mental 
effort  may  be  much  more  difficult  to  realize  for  such  tasks  and  settings.  Finally,  an  important  issue  for  future 
research  pertains  to  the  level  of  control  that  learners  should  be  given  over  task  selection.  The  development  of 
metacognitive  skills  and  higher  order  skills  is  increasingly  seen  as  an  important  goal  of  education,  and  this 
includes  the  ability  to  select  yourself  learning  tasks  that  best  help  to  reach  educational  and  personal  goals.  The 
question  is  not  if  new  learning  tasks  should  be  selected  by  the  teacher  (or  another  intelligent  agent)  or  by  the 
student,  but  how  teachers  can  select  learning  tasks  for  students  who  are  not  yet  able  to  do  this,  how  they  can 
help  students  to  take  more  and  more  responsibility  for  selecting  their  own  learning  tasks,  and  how  they  can  help 
students  with  this  by  giving  advice  and  providing  guidance.  We  will  only  be  able  to  answer  those  questions  if 
we  understand  the  mechanisms  of  good  learning  task  selection. 
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Abstract 

The  demand  for  the  integration  of  technology  into  our  K-12  schools  dictates  the  technology  curriculum 
in  higher  education.  Since  the  evolution  of  new  technologies  is  continuous  and  rapid,  it  is  an  ongoing  challenge 
to  keep  post-secondary  education  current.  Teacher  education  programs  are  no  exception.  Today’s  pre-service 
teachers  are  the  classroom  teachers  of  tomorrow.  They  must  be  prepared  for  the  near  and  distant  futures. 

To  this  end,  faculty  and  staff  in  higher  education  continuously  strive  to  be  on  the  cutting  edge.  There 
is  a constant  alert  for  what's  new  today  and  what’s  coming  tomorrow.  One  is  always  looking  ahead  of  the 
curve  in  order  to  be  prepared  for  the  future.  In  order  to  provide  information  for  the  updating  of  a college 
curriculum,  the  following  study  was  conducted. 


Introduction 

The  purpose  of  the  study  was  to  identify  technology  skills  and  knowledge  that  are  used  and  taught  by 
public  school  teachers  and  others  working  in  the  educational  field.  This  information  was  intended  to  be  used  to 
in  the  planning  and  revision  of  technology  courses  in  higher  education  but  could  also  be  used  for  the  planning 
of  in-service  training  in  the  public  school  systems. 

The  challenge  to  planning  programming  today  is  not  only  being  prepared  for  what  is  to  be  2,  3,  5 or  10 
years  down  the  road,  but  what  is  needed  right  now.  How  do  we  address  the  very  practical  needs  of  today  yet  be 
prepared  for  what’s  coming?  To  ascertain  present  and  future  needs,  public  school  teachers  and  others  working 
in  the  educational  field  were  surveyed.  The  group  included  teachers,  administrators,  librarians,  and  technology 
professionals  currently  enrolled  in  graduate  courses.  Participants  were  asked  about  their  use  patterns  in  the 
following  categories:  hardware,  media/communications,  professional  development,  software,  and  other. 

Rationale 

While  it  is  essential  to  prepare  pre -service  teachers  for  future  technology,  it  is  important  not  to  loose 
sight  of  the  current  tools  of  the  classroom.  Teachers  still  use  the  overhead  projector,  copy  machines, 
audiocassettes,  and  various  other  technologies  not  considered  high-tech.  Since  there  is  great  diversity  in  the 
available  technology  in  classrooms  across  the  country  and  future  teachers  cannot  anticipate  where  they  will  be 
employed,  multiple  contingencies  must  be  anticipated.  Higher  education  teacher  preparation  programs  must 
provide  a wide  range  of  knowledge  and  skills  ranging  from  the  simple  to  the  complex. 

It  is  important  to  be  current  yet  forward-looking  for  a number  of  reasons.  First  of  all,  if  new  teachers 
are  not  prepared  and  comfortable  with  various  technologies  they  have  in  their  classrooms,  they  will  likely  not 
use  them.  If  there  is  a general  feeling  of  incompetence,  they  are  less  likely  to  try  new  things  as  well.  In  a report 
by  the  National  Center  for  Education  Statistics  (U.S.  Dept  of  Education,  2000)  entitled,  Teacher’s  Tools  for  the 
21st  Century,  researchers  found  that  teachers  who  spend  more  time  in  technology  training,  as  opposed  to 
teachers  who  spent  less  time,  felt  well  or  very  well  prepared  to  use  it  for  instruction,  and  teachers’  use  of 
technology  was  related  to  their  training. . . .”  It  is  then  likely  that  there  would  be  impacts  on  the  quantity  and 
quality  of  teaching  and  learning  in  the  classroom. 

The  overall  scope  of  technology  training  in  teacher  education  programs  is  still  unknown.  According  to 
the  National  Education  Association  (NEA,  1998),  “At  least  50%  of  today’s  teachers  have  not  had  adequate 
training  and  technical  assistance  in  the  uses  of  technology.’’  The  NCES  report  found  that  93%  of  public  school 
teachers  in  elementary  and  secondary  education  indicated  that  independent  learning  or  being  self  taught  was  the 
most  frequent  way  they  were  educated  in  technology.  Though  88%  received  training  through  professional 
development,  only  half  of  all  teachers  reported  that  college  and  graduate  work  prepared  them  to  use  technology. 
The  NEA  implies  that  many  teachers  enter  their  profession  with  minimal  technology  training  that  prepares  them 
for  using  it  in  their  classrooms  and/or  curricular  area.  The  report  states:  “Most  teacher  preparation  programs 
provided  by  schools,  colleges,  and  departments  of  education  do  not  have  written,  funded  and  regularly  updated 
technology  plans.’’  The  data  suggests  that  more  technology  programs  in  higher  education  should  be  offered  and 
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updated  to  provide  technology  training  programs  to  pre -service  and  public  school  teachers. 

This  study  included  most  current  technologies,  knowledge  and  skills  which  may  potentially  be  taught 
at  the  college  level.  Although  current  data  from  this  study  may  be  used  to  update  existing  curricula,  it  is  worthy 
to  reiterate  the  changing,  flexible,  and  unpredictable  nature  of  this  field.  Such  studies  and  inquiries  should  be 
made  regularly.  Those  who  plan  programming  should  avail  themselves  of  such  resources  and  data  in  order  to 
update  programs  on  a regularly  scheduled  basis. 

Methodology 


Research  Design 

The  research  design  was  as  a descriptive  study.  In  May  2001,  a graduate  faculty  member  at  a local 
university  piloted  the  original  survey  with  teachers  at  a local  elementary  school  in  the  southwest  Missouri  area. 
One  year  later,  the  survey  was  modified  to  make  it  more  efficient.  The  surveys  were  distributed  via  e-mail 
during  the  summer  of  2002.  A second  distribution  occurred  during  the  early  part  of  the  fall  2002  semester  via 
post  office  mailing  due  to  a lowreturn  rate.  A self-addressed,  stamped  envelope  was  also  included  along  with 
the  survey  and  a letter  reintroducing  the  study.  The  participants  were  asked  to  mail  the  survey  back  within  a 
week. 

Participants 

The  participants  were  graduate  students  in  higher  education  at  a Missouri  university  during  the  summer 
2002  semester.  They  were  enrolled  in  the  following  gradate  level  classes:  Library  Science,  Educational 
Administration,  Introduction  to  Educational  Research,  Educational  Counseling,  Field  Experience,  and  Career 
Development  from  the  Department  of  Guidance  and  Counseling.  Participants  included  a mixture  of  male  and 
female  graduate  students  of  different  ages,  positions,  interests  and  training.  Consent  forms  were  distributed  and 
privacy  insured.  Participants  selected  which  technology  they  use,  which  technology  they  teach  to  others,  which 
technology  they  neither  use  nor  teach  to  others  (therefore  would  require  no  formal  training),  or  which 
participants  would  like  formal  training  on  (See  Table  1). 

Table  1.  Sample  of  instrument  for  first  four  items  of  the  Hardware  category. 


Hardware 

Category 

I use  this 
technology 
myself 

I teach  this 
technology  to 
others 

I neither  use  nor 
teach  this 
technology 

I would  like 
formal  training 
on  this 

1 . Paper  cutter 

2.  Copy 
machine 

3.  Overhead 
projector 

4.  16  mm 
projector 

Results 

Thirty-four  (N)  out  of  63  participants  responded  to  the  survey  via  e-mail  and  postal  mail.  The 
participants  included  teachers,  school  counselors,  case  managers  from  a local  behavior  health  care  clinic,  other 
working  professionals,  and  full-time  graduate  students.  The  majority,  19  out  of  34  of  the  participants  were 
teachers  (Figure  1).  Grade  levels  taught  by  participants  ranged  from  pre-school  to  college.  The  majority,  63%  of 
the  participants,  were  elementary  school  teachers  (Figure  2).  Other  working  professionals  included  a tax  payer 
services  representative  and  an  accountant.  These  were  treated  at  outliers  and  were  not  included  in  the  data 
analysis. 
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Demographics 

Participants  were  counselors,  case 
managers,  teachers,  full-time  students,  and  other 
working  professionals  who  were  graduate 
students  in  higher  education  at  a Missouri 
university.  The  survey  instrument  was  pilot- 
tested  twice  with  a small  group  of  teachers  who 
were  graduate  students  in  the  College  of 
Education.  After  the  second  revision,  surveys 
were  ready  for  distribution.  The  participants 
volunteered  to  be  in  the  study  during  the  summer 
2002  semester  after  the  researcher  presented  the 
study  to  their  graduate  classes.  Surveys  were 
first  distributed  via  e-mail  but,  after  a low 
response  rate,  they  were  redistributed  via  postal 
mailing.  Thirty-four  out  of  60,  or  57%  of  the 
participants,  responded  after  the  second 
distribution  of  surveys. 

The  majority  (56%),  or  19  of  34 
participants,  was  currently  teaching.  Twelve 
of  the  19  taught  in  an  elementary  school.  The 
largest  number  of  participants  (41%),  or  14  of 
34,  received  most  of  their  technology  training 
by  teaching  themselves,  while  only  12%,  or  4 
of  34,  were  trained  in  their  college  program. 
All  four  of  these  had  completed  their 
undergraduate  degrees  within  the  last  five 
years. 

The  demographic  data  regarding 
technology  training  was  not  surprising.  It  is 
apparent  that  most  educators  are  either  self- 


□ Burell  Case 
Managers 

□ Other 
Professional 
Work 


Figure  1.  Job  Position  and  Title  of  Participants. 


Figure  2.  Grade  level  of  teacher  participants 

taught  or  take  advantage  of  workshops  to 
further  their  technical  skills.  It  is  assumed  that 
most  of  these  were  through  in-service 
programs  in  their  schools.  However,  this  was 
not  asked.  They  may  also  have  taken  courses 
from  other  sources  such  as  community 
colleges. 

Other  Data  Analysis 

The  technology  questions  on  the 

survey  were  organized  under  the  following 

main  headings:  Hardware,  Media/  Communications,  Professional  Development,  Software,  and  Other.  Each 

participant  was  asked  to  mark  the  column  that 

, , , 1TTT  . Figure  3.  Sources  of  technology  training. 

mostly  applied  to  them:  “I  Use  This  s & 

Technology  Myself,”  “I  Teach  This  Technology  to  Others,”  “I  Neither  Use  Nor  Teach  This  Technology,”  or  “I 

Would  Like  Formal  Training  On  This  Technology.”  Therefore,  they  could  only  choose  one  column.  The  most 

frequent  responses  from  a total  of  34  are  discussed  in  this  chapter.  The  column  ‘‘I  Teach  This  Technology  to 

Others”  was  not  discussed  due  to  an  insufficient  number  of  responses  under  this  option. 

I Use  This  Technology.. .. 

The  top  three  items  for  each  heading  in  the  “I  use  this  technology”  column  are  summarized  in  Table  2. 
The  results  for  Hardware  were  somewhat  expected  as  all  of  these  are  common  office  and  educational 
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technologies.  In  the  Media/Communications  area  we  also  can  see  some  of  the  most  often  used  media  including 
Bulletin  Boards,  a low-tech  medium.  The  internet  seems  to  play  an  important  part  in  Professional  Development 
as  the  data  suggests  it  is  used  as  a student  and  teacher  resource  as  well  as  for  collaboration.  Word  processing 
ranks  number  one  in  the  Software  category  followed  by  internet  searching  and  finding  technology  resources  and 
spreadsheets  which  tied  for  third  place.  The  Other  category  may  suggest  some  needs  as  troubleshooting 
computers  as  well  as  other  equipment  and  technology  and  diversity  issues  rank  at  the  top.  We  could  surmise 
that  they  spend  a significant  amount  of  time  in  these  activities  and/or  are  concerned  with  these  topics. 

Table  2.  Top  three  data  rankings  for  the  “I  use  this  technology...  ” column. 


Rankings 

Hardware 

Total  of  34 

1 

Copy  machine 

33 

2 

VCR  and  TV- VCR  setups 

32 

3 

Printers 

Media/Communications 

31 

1 

E-mail 

32 

2 

Transparencies 

27 

3 

Bulletin  boards  (design  and  display) 

Professional  Development 

24 

1 

Student  resources  on  the  web 

20 

2 

Teacher  resources  on  the  web 

18 

3 

Collaborative  resources  on  the  Internet 

Software 

18 

1 

Word  processing 

30 

2 

Internet  searching  (search  engines) 

26 

3 

Spreadsheets 

Other 

24 

1 

Troubleshooting  computers 

11 

2 

Technology  and  diversity  issues 

10 

3 

Troubleshooting  a variety  of  equipment 

9 

I Would  Like  Formal  Training... 

Table  3 shows  the  most  frequent  requests  for  formal  training  for  each  of  the  headings.  In  the  Hardware 
area,  thirty-two  percent  (11/34)  wanted  training  on  Desktop  Videoconferencing  and  other  newer  technologies 
such  as  electronic  whiteboards/Smart  Boards  and  interactive  television  (ITV).  This  may  suggest  there  is  an 
awareness  of  their  capabilities  and  potential  applications.  These  technologies  may  or  may  not  be  available  on 
the  job  or  may  not  be  used  due  to  a lack  of  knowledge  and  skills.  Web  site  development,  not  surprisingly,  was  a 
major  area  of  interest  in  Media/Communications.  In  the  Professional  Development  area  it  was  surprising  to  find 
that  Locating  Free  Materials  was  of  high  interest.  Perhaps  this  is  a reflection  of  budgetary  constraints  and  a 
need  for  instructional  materials  and  information.  It  was  also  not  surprising  that  PowerPoint  was  the  #1  Software 
training  request.  In  the  Other  technologies  category  participants  were  primarily  concerned  with  troubleshooting 
and  repairing  all  types  of  technologies.  There  were  also  indicators  of  interest  in  management  issues  such  as 
diversity,  privacy,  and  limited  access.  It  is  apparent  that  educators  in  various  settings  have  concerns  and  needs 
not  only  with  hardware  and  software  operations  but  also  the  ramifications  of  its  effective,  efficient,  and  lawful 
use. 
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Table  3.  Data  rankings  for  “I  would  like  formal  training  on.... 


Hardware 

Rankings 

Total  of  34 

1 

Desktop  videoconferencing 

11 

2 

Interactive  TV  (ITV)  and  Electronic  White  Boards  “Smart” 

10 

3 

Scanner 

8 

Media/Communications 

1 

Web  site  development 

14 

2 

Desktop  publishing  (flyers,  forms,  etc) 

8 

3 

Dry  mounting  and  video  as  a teaching/learning  tool 

5 

Professional  Development 

1 

Locating  free  materials 

9 

2 

Online  training  sites 

7 

3 

Student  resources  on  the  web;  collaborative  resources  on  web 

6 

Software 

1 

Presentation  (PowerPoint,  etc.) 

11 

2 

Interactive  Learning  Systems;  Authoring  (Hyper  Studio,  etc.) 

9 

3 

Databases;  Simulations 

8 

Other 

1 

Troubleshooting  computers  and  variety  of  equipment 

9 

2 

Privacy  and  limited  access  (internet  issues);  equipment  repair 

8 

3 

Technology  and  diversity  issues 

7 

Comparative  Analysis 

In  the  area  of  Media/Communications  we  looked  at  those  who  selected  “Neither  used  nor  taught”  vs. 
“Would  like  training.”  By  examining  the  convergent  and  divergent  points  we  made  a number  of  observations.  In 
the  Desktop  Publishing  area,  the  number  of  “Neither  use  nor  teach”  responses  (7)  was  very  close  to  “Would  like 
training”  (8).  This  could  suggest  that,  even  though  it  is  not  used,  there  are  a nearly  equal  number  of  people 
interested  in  learning  these  skills.  It  could  be  surmised  that  training  in  this  area  would  increase  use  and 
performance. 


Media/Communications 


Figure  4.  Media/Communications  comparisons  of  “I  neither  use  nor  teach”  vs.  “I  would  like  training.” 
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Likewise,  in  the  Media/Communications  section  (Fig.  4),  Web  Site  Development  (16)  was  identified  as 
an  area  that  was  not  used  nor  taught  but  14  requested  training.  Similar  close  relationships  may  be  observed  for: 
Digital  camera.  Scanner,  Desktop  Publishing,  Student  Resources  on  the  Web,  Locating  Free  Materials, 
Spreadsheets,  Databases,  Presentation  Software  (ex.  PPT),  Internet  Searching,  and  Loading  Software.  For  Dry 
Mount,  Video  as  a Teaching  /Learning  Tool,  and  Visual  Literacy  a small  number  of  participants  requested 
formal  training  even  though  most  neither  used  nor  taught  these.  In  this  case,  even  though  the  number  of 
responses  was  low,  it  could  be  that  there  was  little  knowledge  of  these  topics  thus  resulting  in  this  response 
pattern.  It  was  surprising  that  some  interest  was  indicated  perhaps  warranting  a close  look  at  these  areas. 


Professional  Development 


Figure  5.  Professional  Development  comparisons  of  “Neither  use  nor  teach”  vs.  “Would  like  training.” 

Professional  Development  comparisons  (Fig.  5)  for  the  same  response  choices  showed  some  rather 
dramatic  convergent  areas  between  Student  Resources  on  the  Web  and  Locating  Free  Materials  and  divergent 
patterns  with  Online  Training  Sites  and  Technology  Management,  Purchasing,  etc.  One  may  conclude  that,  for 
those  areas  that  closely  correlate  such  Internet  Resources  and  Collaborations,  participants  may  increase  their  use 
given  information  and  training.  On  the  other  hand,  for  those  areas  that  diverge  such  as  in  the  Management  and 
Online  Training  Site  areas,  it  would  seem  that  participants  do  not  engage  in  these  activities  and  are  not 
interested  in  training.  For  example,  24  of  34  neither  use  nor  teach  in  the  area  of  technology 
management/purchasing,  budgeting,  etc  and  only  5 indicated  an  interest  in  learning  more  about  these  areas. 
Given  this  disparity  in  numbers,  it  would  seem  logical  that  this  would  not  be  an  area  of  great  concern  for 
training.  However,  administrators  may  want  to  target  these  small  groups  for  specialized  sessions.  It  would  seem 
reasonable  that  convergent  areas  should  be  addressed  in  training  programs  in  higher  education  and  in -services 
whereas  those  that  diverge  would  not  be  as  critical. 

In  the  Software  category  (Fig.  6)  we  looked  at  actual  use  patterns  and  the  desire  for  training.  Again, 
the  pattern  of  relationships  had  interesting  implications.  Apparently,  high  use  areas  such  as  Finding 
Technology  Resources  thru  the  Library  and  Internet  Searching  are  comfort  zones  which  seem  to  suggest 
training  in  these  areas  is  not  needed.  The  chart  shows  highly  divergent  patterns.  However,  further  study  is 
advised  as  Librarians  may  have  a different  perspective  on  this.  Topics  that  closely  correlate  or  come  together 
on  the  chart  such  as  Presentation  Software  (Ex.  PowerPoint),  Authoring  Software  (ex.  Hyper  Studio), 

Interactive  Learning  Systems,  Simulations,  Evaluation  of  Software,  Evaluation  of  Web  Sites,  Inspiration,  and 
Kidspiration  show  a nearly  equal  number  of  those  who  are  users  and  those  requesting  training.  With  nearly 
equal  numbers  in  these  areas,  it  would  be  logical  to  address  these  areas  in  training  programs. 

There  is  another  interesting  crossover  pattern  in  the  Simulations  area.  More  would  like  training  than 
are  actual  users . Again,  even  though  the  numbers  are  low,  this  seems  to  suggest  that  more  training  and 
information  would  increase  use. 
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Figure  6.  Software  comparisons  of  “I  Use  This  Technology  Myself’  vs.  “I  Would  Like  Formal  Training.” 


The  “Other”  category  (Fig.  7)  included  management  topics  that  are  not  so  much  “technology”  as  issues 
surrounding  its  use,  ex.  troubleshooting,  maintenance,  diversity,  privacy  and  limited  access,  and  purchasing. 
This  chart  shows  the  relationship  of  “Use”  vs.  “Neither”  vs.  “Want  training.”  It  is  evident  that  the  majority  of 
participants  are  not  involved  in  these  activities.  However,  there  is  seems  to  be  a relationship  between  “Use” 
and  “Want  training.”  Two  coordinating  points  are  in  the  “Troubleshooting  equipment”  area.  It  appears  that 
users  also  want  to  be  able  to  troubleshoot.  On  the  other  hand,  there  are  more  who  indicated  they  are  not 
“Users”  of  Internet  Privacy  and  Limited  Access  topics  than  “Would  like  training.”  Perhaps,  given  information 
and  training,  more  would  concern  themselves  with  these  issues. 


Figure  7.  Comparisons  of  “Use,”  “Neither  use  nor  teach”  and  “Would  like  training”  in  the  Other  category. 

Data  Summary 

The  top  three  choices  in  each  category  are  a mixture  of  old  and  new  technologies  and  topics.  From  a 
sample  population  of  34,  note  the  actual  counts  in  parentheses  within  the  table. 

Table  4.  Top  three  rankings  for  the  “I  use  this  technology”  response. 


Category 

Ranked  #1 

Ranked  #2 

Ranked  #3 

Hardware 

Copier  (33) 

VCR  setup  (32) 

Printers  (31) 

Media/ 

Communications 

E-mail  (32) 

Transparencies  (27) 

Bulletin  Boards  (24) 

Professional 

Development 

Student 
Resources  (20) 

Teacher  & Collaborative 
Resources  (tied- 18) 

Locating  Free 
Materials  (16) 

Software 

Word  processing  (30) 

Internet  searching 
(Search  engines)  (26) 

Spreadsheets 
& Finding  technology 
Resources  through  the 
Library  (tied-24) 

Other 

Troubleshooting 
Computers  (11) 

Technology  & 
Diversity  (10) 

Troubleshooting 
other  equipment  (9) 

Table  5.  Top  three  rankings  for  the  “I  would  like  formal  training”  category. 


Category 

Ranked  #1 

Ranked  #2 

Ranked  #3 

Hardware 

Desktop 

Videoconferencing!  11) 

Interactive  TV  & 

Electronic  white  boards  (tied-10) 

Scanner (3) 

Media/Communications 

Web  site 

Development  (14) 

Desktop  Publishing  (8) 

Dry  Mount  & 

Video  as  teaching/learning 
tool  (tied-5) 
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Professional 

Development 

Locating  Free 
Materials  (9) 

Online  Training  Sites  (7) 

Student  & 
Collaborative 
Resources  (tied-6) 

Software 

PowerPoint  (11) 

Authoring  software.  Loading, 
Interactive  Learning  Systems  (9) 

Databases  & 
Simulations  (tied-8) 

Other 

Troubleshooting  all 
Technology  (9) 

Privacy  and  limited  access  & 
Equipment  repair  (tied- 8) 

Technology  and 
Diversity  Issues  (7) 

Table  6.  Data  summary  comparing  present  use  and  areas  where  training  is  desired. 


Hardware 

Media/ 

Communications 

Professional 

Development 

Software 

Other 

Use  Now 

Copier 

E-mail 

Student  Resources 

Word 

processing 

Troubleshooting 

Computers 

Want  Training 

Video 

Conferencing 

Web 

Development 

Locating  Free 
Materials 

PowerPoint 

Troubleshooting 
all  technology 

Discussion 

It  is  important  to  note  that  technologies  that  are  used  often  should  also  be  considered  in  training 
programs.  Although  this  population  was  not  asked  if  they  had  training  in  these  areas,  one  could  reasonably 
assume  not  all  have  had  in-depth  training  in  even  the  most  commonly  used  technologies  such  as  word 
processing.  Versions  and  needs  change  as  well  as  operating  systems  and  uses.  A well-rounded  program, 
whether  in  higher  education  or  an  in-service,  would  address  all  areas  of  potential  need.  It  is  also  important  to 
note  that.  We  must  remember  that  techniques,  presentation  tips,  design  skills,  and  foundation  principles  do  not 
change  over  time  and  should  be  included  in  all  programs  that  educate  teachers.  In  higher  education  it  is 
especially  important  to  teach  the  foundational  theories  of  the  field  of  instructional  technology  as  applied  to 
present  tools  as  well  as  those  on  the  horizon. 

Although  the  data  in  the  areas  referring  to  older  technologies  may  show  little  interest  in  training  in 
these  areas,  it  is  important  to  reme  mber  that:  1)  initial  training  in  the  technology  may  have  been  inadequate  (few 
know  more  than  basic  skills  in  word  processing  or  how  to  create  a simple  one-layer  transparency);  2)  software 
is  continuously  being  updated  requiring  re -training;  and  3)  new  skills  and  theories  in  the  application  of 
technology  require  continuous  education  for  our  teaching  professionals. 

Summary 

The  purpose  of  this  survey  was  to  gather  data  concerning  the  knowledge,  skills,  and  technology 
teachers  are  using  in  the  K-12  public  school  setting.  Demographic  data  showed  56%  or  19  of  the  34 
respondents  were  K-12  teachers  followed  by  school  counselors,  full-time  students,  case  manages  for  a 
behavioral  institute,  and  two  undeclared.  All  were  graduates  in  university  courses.  Questions  focused  on  what 
they  use  themselves,  which  they  teach  to  others,  which  they  neither  use  nor  teach,  and  those  for  which  they 
would  like  training.  This  information  could  then  be  used  to  determine  changes  and  updates  to  the  technology 
curriculum  for  pre -service  teachers  at  the  university  level.  Although  this  was  a limited  survey,  it  did  suggest 
some  possible  trends  and  relationships  which  seem  to  be  in  alignment  with  current  beliefs. 

Results  were  predictable  in  some  ways.  The  most  used  technologies  were  the  most  available  in 
schools:  copy  machines,  overhead  transparency,  paper  cutters,  etc.  Conversely,  those  that  were  identified  as 
“Neither  use  nor  teach”  were  either  older  and/or  no  longer  available  (16  mm  projectors),  managed  by  a 
specialist  or  coordinator  (networks,  budgets,  etc.),  or  so  new  they  were  not  likely  available  (electronic  white 
boards).  The  latter  was  evident  in  the  “I  would  like  training . . . .”  response  as  they  were  mostly  the  newer 
technologies.  The  “Other”  category  revealed  a desire  for  moie  troubleshooting  knowledge  and  skills. 

Several  recommendations  are  being  made.  Since  67%  of  the  participants  indicated  most  of  their 
technology  skills  were  self-taught  or  gained  through  workshops,  it  is  evident  that  higher  education  must 
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continue  to  be  diligent  in  the  training  of  our  pre-service  teachers.  Until  that  portion  of  the  workforce  with 
limited  knowledge  and  skills  retires  or  is  trained  and/or  the  current  generation  becomes  the  majority,  there  will 
be  a need  to  continue  with  broad-spectrum  pre-service  technology  training  programs. 

It  would  also  behoove  us  to  remember  that  those  established  technologies  and  media  which  have  a 
proven  track  record  as  being  effective  and  reliable  (bulletin  boards  and  transparencies)  will  continue  to  be 
integral  in  the  education  of  our  students.  As  the  data  showed,  teachers  in  the  classroom  are  aware  of  this  and  are 
requesting  training  in  these  areas.  There  are  many  subtleties  to  the  efficient  production  and  use  of  these  media 
which  are  not  likely  learned  on  the  job.  Visual  literacy  skills  as  well  as  production  skills  are  required  for 
classroom  teachers  to  communicate  with  students  in  an  efficient,  clear  manner  regardless  of  the  technology 
being  used.  Teachers  must  be  able  to  create  and  interpret  visually  in  order  to  create  effective  instructional 
materials.  Instructional  design  and  technology  faculty,  art  teachers,  and  communication  specialists  in  many 
areas  may  assist  with  this  very  important  task  of  training  both  visual  and  verbal  communication  skills. 

This  maybe  accomplished  in  several  ways.  Partnerships  of  various  types  including  those  with 
universities,  school  districts,  community  colleges,  the  private  sector,  etc.  may  provide  the  financial  resources 
and  other  opportunities  required  in  basic  and  advanced  training  programs  at  all  levels.  With  continuing 
restrictions  on  budgets  in  any  one  institution,  it  would  seem  that  these  kinds  of  collaborations  would  be 
mutually  beneficial. 

In  order  to  keep  current  with  the  changes  in  the  field  of  education  and  technology  it  is  important  to 
conduct  surveys  such  as  this  as  an  assessment  activity.  Items,  categories,  and  needs  will  change  continuously. 
In  as  little  as  two  years  this  picture  could  change.  Therefore,  it  is  recommended  that  this  type  of  evaluation  be 
used  at  all  levels  of  education  including  post-secondary.  This  would  also  provide  important  data  for 
accreditation  purposes.  It  would  also  be  helpful  to  ascertain  not  only  use  pattern  but  also  what  they  would  like 
training  on  by  allowing  participants  to  select  more  than  one  response  choice  instead  of  only  the  one. 

Conclusion 

In  a report  by  the  National  Center  for  Education  Statistics  (U.S.  Dept,  of  Education,  2000)  entitled, 
“Teachers’  Tools  for  the  21st  Century,”  researchers  found  that  teachers  who  spent  more  time  in  technology 
training,  as  opposed  to  teachers  who  spent  less  time,  felt  well  or  very  well  prepared  to  use  it  for  instruction  and, 
teachers’  use  of  technology  was  related  to  their  training....”  Our  challenge  as  those  who  prepare  educators  for 
the  classroom  is  to  be  aware  of  the  changing  needs  in  learning  and  technology  and  how  this  may  impact  our 
programs  while  establishing  practices  based  on  sound  theories  and  techniques.  Diligent  observation  and 
feedback  is  required  to  meet  this  challenge. 
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Introduction 

Utilization  of  technology  in  secondary  schools  is  varied  and  depends  on  the  training  and  interest  of  the 
individual  instructors.  Even  though  technology  has  advanced  way  beyond  its  utilitarian  roots  of  being  viewed 
solely  by  educators  as  a useful  machine  for  teachers  to  key  exams  and  worksheets  on,  there  are  still  many 
secondary  educators  who  still  only  view  it  as  such.  These  educators  have  not  recognized  the  growing  role  that 
technology  is  taking  on  in  today’s  classrooms  of  being  a cognitive  tool  that  when  partnered  with  learning  theory 
can  help  educators  enhance  learning  and  maximize  the  learning  potential  of  their  students.  As  a teacher 
education  professor,  I was  experiencing  this  lack  of  acceptance  of  technology’s  new  role  in  my  students.  When 
I asked  them  if  they  utilized  technology  in  their  classrooms,  they  would  say  that,  of  course  they  do.  Then,  when 
I would  ask  them  how  they  utilized  it,  they  would  say,  to  type  exams,  worksheets,  and  have  their  students  type 
papers.  They  do  not  utilize  it  as  a cognitive  tool  to  enhance  learning.  Concerned  with  this  phenomenon,  I 
proposed  to  find  and  research  teachers  who  did  recognize  and  utilize  technology  as  a cognitive  tool  to  enhance 
learning  for  their  students.  My  search  resulted  in  an  investigation  of  instructors  who  believed  in  and 
implemented  Brain-Based  Learning  with  technological  support  and  had  positive  results.  The  primary  question 
guiding  this  study  was,  “How  is  Brain-Based  Learning  with  technological  support  being  implemented  by 
selected  teachers  in  their  classrooms?’’  The  corollary  questions  included:  What  is  the  nature  of  the 
environment,  that  is,  the  classroom,  in  a Brain-Based  Learning  environment  utilizing  technology?  What  is  the 
role  of  the  instructor  in  a Brain-Based  Learning  classroom  utilizing  technology?  What  is  the  role  of  the  students 
in  a Brain-Based  Learning  classroom  utilizing  technology?  What  are  the  roles  of  the  school  administrator  and 
technology  coordinator  in  the  Brain-Based  Learning  classroom  utilizing  technology?  and  What  are  the  problems 
encountered  by  teachers  when  utilizing  Brain -Based  Learning  with  technological  support  in  the  classrooms  and 
how  can  these  problems  be  overcome? 


Literature  Review 

Brain-Based  Learning  Research 

Researchers,  such  as  Geoffrey  Caine  and  Renate  Nummela  Caine,  recognized  that  the  brain  has  a 
virtually  inexhaustible  capacity  to  learn,  and  that  each  healthy  human  brain,  irrespective  of  a person’s  age,  sex, 
nationality,  or  cultural  background,  comes  with  the  following  features:  the  ability  to  detect  patterns  and  to 
make  approximations;  a phenomenal  capacity  for  various  types  of  memory;  the  ability  to  self -correct  and  learn 
from  experience  by  way  of  analysis  of  external  data  and  self -reflection;  and  an  inexhaustible  capacity  to  create. 
They  could  not,  though,  understand  why,  if  everyone  has  these  features,  we  are  struggling  with  the  ability  to 
educate.  Caine  and  Caine  (1994)  found  the  answer  to  this  was  that  educators  did  not  know  and  understand  the 
complexity  and  elegance  of  the  way  the  brain  learns,  especially  when  it  is  functioning  optimally.  Their  research 
presented  this  information  and  how  it  could  be  utilized  to  enhance  learning  for  all  students.  For  example, 
teachers  need  to  provide  learning  activities  and  experiences  that  immerse  learners  in  curricular  content  and 
context,  such  as  encouraging  them  to  talk,  listen,  read,  view,  act,  and  value  what  is  being  learned  (1994).  In 
other  words,  they  needed  to  implement  a brain -based  education  for  all  students.  A brain -based  education, 
according  to  Caine  and  Caine  (1994),  involved  designing  and  orchestrating  lifelike,  enriching,  and  appropriate 
experiences  for  learners  and  ensuring  that  students  process  experiences  in  such  a way  as  to  increase  the 
extraction  of  meaning.  This,  they  indicated,  could  be  accomplished  by  providing  a variety  of  learning  activities 
and  projects,  and  choices  where  those  activities  and  projects  would  take  place. 

Another  researcher,  Howard  Gardner  (1999),  added  to  the  concept  of  Brain-Based  Learning  by 
establishing  his  theory  of  “multiple  intelligences,’’  a theory  that  indicates  that  there  is  not  just  one  form  of 
intelligence  based  on  verbal  and  reasoning  abilities,  but  rather  eight  different  intelligences,  each  having  a 
unique  neurological  pattern  and  course  of  development.  The  eight  intelligences  include:  linguistic,  musical, 
logical- mathematical,  spatial,  bodily -kinesthetic,  interpersonal,  intrapersonal,  and  naturalistic.  Gardner  (1999) 
believed  that  the  educational  system  of  the  1970s  favored  children  who  were  skilled  at  reading  and  writing,  and 
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did  not  address  a student  who  was  considered  “poor”  in  school  due  to  his/her  limited  reading  and  writing  skills, 
but  who  could,  in  fact,  operate  a film  projector  without  instruction,  which  takes  spatial  intelligence.  In  other 
words,  Gardner  felt  that  there  were  too  many  children  just  like  this  one  whose  educational  needs  were  not  being 
met  due  to  the  narrow  view  of  intelligence.  To  meet  all  children’s  learning  needs  teachers  needed  to  present  a 
variety  of  learning  activities,  such  as  individual  and  group  projects,  apprenticeships,  and  hand -on  activities, 
which  addressed  all  eight  of  the  intelligences. 

Lastly,  a researcher  and  educator,  Kathy  Nunley  (2001),  who  was  concerned  that  teachers  did  not  have 
the  education  and  training  to  help  them  find  new  ways  to  meet  the  needs  of  their  increasingly  diverse  groups  of 
students  with  a wide  variety  of  abilities,  cultures,  and  languages,  began  to  research  a way  to  provide  these  for 
them.  Her  research,  based  on  knowledge  of  the  brain  and  Gardner’s  Multiple  Intelligences  Theory,  resulted  in 
the  development  of  a curriculum  entitled  “The  Layered  Curriculum.”  This  curriculum  involves  presenting 
students  with  learning  in  three  distinct  layers.  Each  layer  represents  a different  depth  of  study  in  a topic  or  unit 
of  learning  as  it  is  geared  toward  a different  kind  and  level  of  learning.  Students  can  choose  how  deep  they 
wish  to  examine  a topic  and  which  way  best  fits  their  learning  style  and  ability,  thereby  choosing  the  grade  (A- 
F)  they  will  earn  as  well.  Also,  there  is  a broad  range  of  learning  tools  available  to  the  student  to  assist  in 
completing  their  assignments.  These  tools  include  a wide  use  of  technology,  from  taped  lectures  to  computer 
programs. 

The  Role  of  Technology  in  the  Learning  Process 

Technology  has  long  been  a support  tool  in  education  (Lockard  & Abrams,  2001).  Its  functions  are 
deeply  embedded  in  school  administrative  and  instructional  environments.  Classrooms,  too,  are  places  where 
technology  is  found.  From  movie  and  overhead  projectors,  televisions,  VCRs,  and  tape  recorders,  to  DVDs  and 
computers,  teachers  have  long  found  technology  to  be  a tool  that  enhances  content.  Its  use  in  the  classroom  has 
led  to  technology  being  referred  to  as  a cognitive  tool.  Jonassen  and  Reeves  (1996)  stated  that  “cognitive  tools 
refer  to  technologies,  tangible  or  intangible,  that  enhance  the  cognitive  powers  of  human  beings  during  thinking, 
problem  solving,  and  learning.  Written  language,  mathematical  notation,  and,  most  recently,  the  universal 
computer  are  examples  of  cognitive  tools”  (p.  693).  Jonassen  went  on  to  describe  technology  as  “mindtools.” 
The  idea  of  technology  as  mindtools  parallels  the  newer  view  of  learning  that  has  changed  from  being  viewed 
solely  as  a passive  activity,  where  learners  sit  at  desks  listening,  taking  notes,  studying,  and  taking  tests  to 
measure  the  learning  that  has  taken  place,  to  being  viewed  as  an  active  process,  where  students  actively  are 
engaged  in  the  learning  process  and  in  constructing  their  own  knowledge.  This  active  view  of  learning  often  is 
called  constructivism  and  it  “is  grounded  in  the  research  of  Piaget,  Vygotsky,  the  Gestalt  psychologists,  Bartlett, 
and  Bruner  as  well  as  the  educational  philosophy  of  John  Dewey”  (Woolfolk,  2001,  p.  329).  Constructivism  is 
defined  by  Woolfolk  (2001)  as  “a  view  that  emphasizes  the  active  role  of  the  learner  in  building  understanding 
and  making  sense  of  information”  (p.  329).  Constructivist  learning,  as  an  active  student  centered  learning 
process,  has  several  characteristics.  They  are  that  it  should  involve  “complex,  challenging  learning 
environments  and  authentic  tasks;  social  negotiation  and  shared  responsibility  as  a part  of  learning;  multiple 
representations  of  content;  understanding  that  knowledge  is  constructed;  and  student-centered  instruction” 
(Woolfolk,  2001,  pp.  334-336).  Mindtools  are  cognitive  tools  that  students  utilize  as  they  are  engaged  in  a 
constructivist  learning  environment. 

Brain-Based  Learning  is  student  centered  learning  that  utilizes  the  whole  brain  and  recognizes  that  not 
all  students  learn  in  the  same  way.  It  is  also  an  active  process  where  students  are  actively  engaged  in 
constructing  their  own  knowledge  in  a variety  of  learning  situations  and  contexts  (Caine  & Caine,  1994). 
Mindtools  are  the  cognitive  tools  that  can  support  Brain-Based  Learning. 

Method 


Participants 

The  participants  for  this  case  were  unique  or  “based  on  unique,  atypical,  perhaps  rare  attributes  or 
occurrences  of  the  phenomenon  of  interest”  (Merriam,  1998,  p.  62).  The  six  teachers  were  unique  in  that  they 
had  embraced  Brain-Based  Learning  with  technological  support  and  were  utilizing  it  in  their  classroom 
teaching.  They  were  also  a network  sample  or  a “sample  that  is  based  on  each  participant  or  group  of 
participants  referring  you  to  other  participants”  (Merriam,  1998,  p.  63).  They  were  chosen  based  on 
recommendations  by  the  principal,  as  they  had  studied  and  embraced  Brain-Based  Learning  with  technological 
support  and  had  implemented  its  use  in  their  classrooms.  The  six  teachers  were  from  the  following  disciplines: 
health  education,  physics,  engineering  graphics/CAD/Manufacturing,  French,  history,  and  information  systems 
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The  administrator  constituted  a unique  sample  because  he  was  a strong  advocate  of  Brain-Based 
Learning  with  technological  support  and  was  instrumental  in  bringing  BBL  to  his  school.  He  had  taken 
numerous  workshops  on  Brain-Based  Learning,  had  conducted  his  own  workshops  on  “Creating  a Brain- 
Friendly  Instructional  Climate”  for  local  educators,  and  had  implemented  his  concept  of  a Brain -Friendly 
Instructional  Climate  at  the  high  school  for  the  2000-2001  academic  year. 

The  technology  coordinator  constituted  a unique  sample  because  he  was  the  only  technology 
coordinator  at  the  high  school.  He  worked  closely  with  the  principal  and  the  teachers  mentioned  above  to  assist 
in  the  implementation  of  Brain -Based  Learning  with  technological  support. 

Data  Collection 

Data  collection  employed  the  strategies  of  interviewing,  observing,  and  completing  a checklist.  The 
interviews  involved  meeting  with  each  of  the  six  teachers,  the  administrator,  and  the  technology  coordinator  and 
having  them  discuss  their  individual  roles  in  Brain-Based  Learning  with  technological  support,  how  such 
learning  can  be  implemented,  the  problems  that  were  encountered,  and  possible  solutions  to  these  problems. 

The  observations  involved  observing  the  six  teachers  as  they  were  implementing  Brain-Based  Learning  with 
technological  support.  Lastly,  the  checklist  involved  recording  the  Brain-Based  methods  utilized  by  teachers  in 
the  classrooms  during  the  observations. 


Results 

The  data  collected  from  interviews,  observations,  checklists,  and  questionnaires  were  analyzed  and 
consolidated,  and  the  findings  organized  and  presented  as  they  relate  to  the  research  questions. 

Corollary  Question  1 

Corollary  Question  1 asked,  “What  is  the  nature  of  the  environment,  that  is,  the  classroom,  in  a brain - 
based  learning  environment  utilizing  technology?”  The  data  indicated  that  the  nature  of  the  classroom  in  a 
brain-based  learning  environment  is  one  that  is  active  and  learner-centered.  It  is  visually  appealing,  warm,  and 
conducive  to  interaction,  whether  it  be  student-to-student  or  student-to-teacher.  It  is  an  environment  where 
students  are  not  just  sitting  passively  taking  notes  and  listening  to  a teacher  lecture.  It  is  where  students  are 
actively  engaged  in  learning,  interacting  with  a variety  of  learning  tools,  their  peers,  and  the  teacher.  The 
teacher  is  not  only  seen  in  the  front  of  the  classroom,  but  is  frequently  seen  walking  around  the  classroom 
talking  to  students  and  helping  them  with  problems,  answering  questions,  and  offering  feedback.  The  BBL 
classroom  is  an  environment  where  as  students  enter  through  the  classroom  doors,  they  are  transported  into  the 
world  of  the  subject  being  taught.  Lastly,  it  is  an  environment  that  is  computer-based.  All  of  the  classrooms 
have  computers  in  them  or  have  access  to  a computer  lab,  where  students  frequently  are  actively  engaged  in 
working  through  learning  modules,  in  doing  research,  or  in  a variety  of  creative  endeavors. 

Corollary  Question  2 

Corollary  Question  2 asked,  “What  is  the  role  of  the  instructor  in  a Brain-Based  Learning  classroom 
with  technological  support?”  The  data  indicated  that  the  role  of  the  instructor  in  a Brain-Based  Learning 
classroom  with  technological  support  is  as  a guide  and  facilitator  to  the  students  as  they  are  actively  engaged  in 
the  learning  process.  In  this  role  the  instructor  provides  direction,  answers  questions,  guides  collaborative 
problem  solving,  and  offers  feedback.  The  instructor  has  as  his/her  philosophy  the  knowledge  that  all  students 
are  individuals  with  individual  brain  dominance  and  intelligence(s),  information  processing  abilities,  and 
learning  styles,  and  that  learning  needs  to  be  adapted  so  that  all  students  can  have  learning  experiences  that 
enable  them  to  learn  to  the  best  of  their  abilities.  This  can  be  accomplished  through  varying  activities,  learning 
tools,  and  approaches. 

Corollary  Question  3 

Corollary  Question  3 asks,  “What  is  the  role  of  the  students  in  a Brain-Based  Learning  classroom  with 
technological  support?”  The  data  indicated  the  role  of  the  students  in  a Brain-Based  Learning  classroom  with 
technological  support  is  that  of  active  learners  engaged  in  the  learning  process,  learning  to  the  best  of  their 
individual  abilities. 

Corollary  Question  4 

Corollary  Question  4 asked,  “What  are  the  roles  of  the  school  administrator  and  technology 
coordinator  in  the  Brain-Based  Learning  classroom  with  technological  support?”  The  role  of  the  administrator 
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committed  to  Brain-Based  Learning  with  technological  support  is,  according  to  the  principal,  “to  try  to  create  a 
risk  free  climate  where  a teacher  is  not  afraid  to  try  something  new  or  different,  and  to  provide  the  support 
elements  for  that  teacher  who  wants  to  try  it.  If  a teacher  wants  to  try  something  with  technology,  then  they 
have  to  have  the  technology,  they  have  to  have  the  skills,  and  they  have  to  have  the  training.  So,  as  an 
Administrator,  my  responsibility  is  to  facilitate  that  climate  so  that  that  teacher  can  take  advantage  of  those 
things  so  they  can  utilize  it  and  have  a better  opportunity  for  success.” 

The  role  of  the  technology  coordinator  in  a Brain -Based  Learning  environment  with  technological 
support  is,  according  to  the  technology  coordinator,  someone  who  wears  “many  hats.”  He  oversees  all 
hardware  and  software  purchasing  and  budgeting  for  both  the  administrative  and  the  teaching  sides  of  the 
building,  and  he  coordinates  all  the  technology  within  the  building.  One  of  the  additional  hats  the  technology 
coordinator  wears  is  that  of  helping  teachers  who  want  to  utilize  technology  in  the  learning/teaching  process. 

Corollary  Question  5 

Corollary  Question  5 asked,  “What  are  the  problems  encountered  by  teachers  when  implementing 
Brain-Based  Learning  with  technological  support  in  the  classrooms  and  how  can  these  problems  be  overcome?” 
The  data  indicated  that  the  problems  encountered  by  the  teachers  involved  in  the  implementation  of  Brain  - 
Based  Learning  with  technological  support  in  classrooms  were  of  two  types.  They  were:  (1)  technical 
problems  and  (2)  student  problems. 

The  technical  problems  encountered  by  the  teachers  when  implementing  Brain-Based  Learning  with 
technological  support  were:  ( 1)  the  network  was  down,  (2)  the  CD  was  not  working,  (3)  there  was  a lack  of 
color  printing  capabilities,  (4)  the  font  was  too  small,  and  (5)  there  was  software  incompatibility.  These 
problems  were  observed  to  be  intermittent  and  none  interrupted  the  teaching/learning  process  for  very  long.  In 
most  cases,  the  instructor  was  able  to  work  around  these  technological  problems  by  having  an  alternate  method 
for  accomplishing  a goal  or  changing  to  another  task  for  the  class. 

The  student  problems  encountered  by  teachers  in  using  technology  to  support  Brain-Based  Learning 
were:  (1)  the  students  were  “surfing”  the  web,  (2)  there  were  multiple  simultaneous  questions,  and  (3)  there 
were  noisy  students.  As  with  the  technical  problems,  these  problems  were  observed  to  be  intermittent  and  none 
interrupted  the  teaching/learning  process  for  very  long.  In  most  cases,  the  instructor  was  able  to  work  around 
these  technological  problems  by  having  an  alternate  method  for  accomplishing  a goal  or  changing  to  another 
task  for  the  class. 


Discussion 

During  the  months  of  collecting  and  analyzing  data  and  reporting  the  findings  of  this  study,  as  the 
researcher,  I found  that  my  personal  beliefs  and  thoughts  on  secondary  education  and  how  it  could  be  improved 
to  meet  the  needs  of  our  increasingly  diverse  high  school  student  population  were  reinforced  and  strengthened.  I 
would  like  to  discuss  some  of  the  more  important  ones. 

First,  learning  for  secondary  students  can  no  longer  be  considered  a passive  activity  where  teachers 
lecture  and  students  sit  in  desks  taking  notes,  studying,  and  taking  tests.  Instead,  learning  should  be  viewed  as  a 
constructivist  process,  an  active  and  engaging  process,  one  that  emphasizes  the  active  role  of  learners  in 
building,  understanding  and  making  sense  of  information  and  reality  (Woolfolk,  2001).  As  active  learners, 
students  may  work  independently  and,  at  times,  cooperatively  on  a variety  of  learning  activities  utilizing  many 
different  learning  materials  and  modes  of  instruction.  Their  teachers  in  the  learning  process  no  longer  just  stand 
in  front  of  the  classroom.  They  now  move  about  it,  stop  by  students  and  answer  questions,  help  the  students 
problem  solve,  and  offer  them  feedback.  They  are  viewed  as  guides,  facilitators,  managers,  supervisors,  and 
models. 

Second,  each  student  processes  information  differently  and  learns  differently,  so  a “one-size -fits -all” 
curriculum  is  no  longer  the  way  to  plan  instruction.  As  the  principal  in  this  study  stated,  “We  have  students  that 
walk  into  this  building  and  it  doesn’t  make  a difference  what  we  do,  they’re  going  to  perform  at  high  levels. 

But  it  is  public  education’s  responsibility  to  deliver  the  educational  services  to  all  of  the  students  who  walk  in, 
not  just  the  smart  ones,  not  just  the  pretty  ones,  not  just  the  ones  who  get  along  with  people,  all  the  students  who 
walk  into  this  building.  In  order  to  do  that,  from  my  experience  and  from  what  I’ve  learned,  and  things  like 
that,  the  way  to  do  that  is  to  create  that  brain  friendly  environment,  that  environment  that  has  mutual  trust  and 
respect,  where  students  feel  connected,  where  their  whole  basic  needs  are  met  so  they  can  move  forward  from 
the  emotional  to  the  intellectual  and  things  like  that.  That’s  the  only  way  to  do  it.”  This  means  that  teachers 
need  to  change  their  view  of  teaching  and  what  it  involves. 

Third,  teachers  in  their  new  roles  and  with  their  new  views  of  teaching  and  the  learning  process  need  to 
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recognize  that  this  involves  no  longer  depending  solely  on  lectures  and  books  to  deliver  knowledge.  They  need 
to  utilize  a variety  of  cognitive  tools,  technology  among  them  (Jonassen  & Reeves,  1996).  As  Howard  Gardner 
(1999)  states:  “One  fact  that  will  make  individually  configured  education  a reality  in  my  lifetime:  the  ready 
availability  of  new  and  flexible  technologies.  Technology  can  be  ‘smart’:  It  can  adjust  on  the  basis  of  earlier 
learning  experiences,  ensuring  that  a student  receives  lessons  that  are  optimally  and  individually  crafted”  (pp. 
153-154). 

In  addition  to  being  a tool  that  can  adapt  to  individual  learners,  technology  is  a tool  that  creates  new 
opportunities  for  curriculum  and  instruction  by  bringing  real-world  problems  into  the  classroom  for  students  to 
explore  and  solve.  For  example,  students  no  longer  just  read  about  people,  places  and  events.  They  can,  via 
technology,  visit  specific  parts  of  the  environment  that  they  have  studied  to  explore  them  more  fully,  to  test 
ideas,  and  to  receive  feedback  (Bransford,  Brown,  and  Cocking,  1999).  They  can  work  on  projects 
independently  or  cooperatively  with  technological  components  including  databases,  spreadsheets,  semantic 
networks,  expert  systems,  multimedia/hypermedia  construction  software,  computer-based  conferencing, 
collaborative  knowledge  construction  environments,  computer  programming  languages,  and  microworlds 
(Jonassen  & Reeves,  1996).  They  can  utilize  “visualization  and  modeling  software  that  is  similar  to  the  tools 
used  in  non-school  environments,  increasing  their  understanding  and  the  likelihood  of  transfer  from  school  to 
non-school  settings”  (Bransford,  Brown,  and  Cocking,  1999,  p.  195),  or  in  other  words,  students  can  acquire 
knowledge  of  tools  and  experience  in  school  that  can  be  transferred  to  the  real  world  environment  of  work. 

Fourth,  this  study,  examined  a high  school,  its  administrator  (principal),  and  its  technology 
coordinator,  all  of  whom  had  embraced  Brain-Based  Learning,  were  utilizing  it  in  their  classrooms  to  meet  the 
varied  learning  needs  of  their  students,  and  were  encountering  very  few  problems  in  the  process.  It  provides 
some  of  the  needed  supporting  evidence  for  Brain-Based  Learning  with  technological  support  to  be  viewed  as  a 
viable  alternative  to  the  traditional  “one -size -fits -all”  thinking  present  in  many  of  today’s  classrooms. 
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Abstract 

This  study  provides  an  in-depth  description  of  adult  learners  ’ perceptions  and  experiences  using 
laptop  computers  in  higher  education  institutions.  The  study  also  examines  the  influence  on  human  beings’ 
cognitive  activity  by  using  laptops.  The  results  are  organized  into  seven  themes  and  four  classifications:  usage, 
benefits,  limitations,  and  cognitive  impacts. 


Introduction 

Portability  characteristic,  wireless  connection,  and  sophisticated  functions  of  laptop  computers 
provide  users  more  mobility  and  freedom  of  choice.  Using  laptop  computers  is  leading  to  a new  kind  of 
learning  format.  Educators  employ  the  term  ‘M-learning”  to  emphasize  cognitive,  pedagogical,  and  social 
aspects  of  mobile  technology-using  for  educational  possibilities.  Many  schools  and  corporations  such  as 
Mircrosoft,  Toshiba,  Apple,  Netschools,  etc.,  have  set  up  projects  to  promote  the  use  of  laptop  computers  for 
educational  use.  Laptop  computers,  along  with  other  portable  technology,  have  become  tokens  of  a new 
learning  format. 

While  technological  advances  have  made  ‘Anytime,  anywhere”  learning  feasible,  few  studies  have 
been  conducted  to  examine  in-depth  views  of  the  learners  who  are  using  laptop  computers.  If  we  adopt 
Vygotsky’s  (1978)  socio -historical  perspective  that  using  new  tools  gives  rise  to  new  social  structures  and 
mental  structures,  we  can  further  conclude  that  learning  and  cognition,  which  are  based  on  mental  structures  and 
are  reflection  of  social  existence,  will  also  change  correspondingly.  Therefore,  it  would  be  valuable  to  study 
such  changes  in  mobile  learners  during  the  transitional  period  from  e-learning  to  m-learning. 

Research  Questions 

The  purpose  of  this  study  is  to  examine  the  experiences  of  adult  users  who  use  laptop  computers  to 
support  their  daily  learning  activities  in  higher  education  institutions.  Researchers  are  interested  in  knowing 
how  laptop  computers  serve  as  mobile  learning  tools.  Researchers  attempt  to  provide  an  in-depth  description  of 
learners’  perspective  and  experiences  about  integrating  these  mobile  devices  into  educational  use. 

Four  specific  questions  are:  How  have  laptops  been  used  in  academic  activities  ? What  benefits  do 
learners  get  by  using  laptops?  How  laptop  computers  change  learners’  cognitive  activities?  What  problems 
exist  during  using  laptops  for  educational  use?  The  results  from  the  study  may  provide  implications  for  more 
effective  technology  services  and  mobile  learning  pedagogy,  design,  and  development. 

Literature  Review 

Sophisticated  laptop  and  wireless  technology,  as  alternatives  of  desktop  and  wired  technology 
respectively,  have  gained  much  importance  in  the  recent  years.  Inspired  by  the  improvements  in  portable 
computing  technology  and  examples  of  successful  pilot  programs,  more  and  more  schools  and  corporations 
have  initiated  projects  to  promote  use  of  laptop  computers  (Belanger,  2002).  Laptop  computers,  just  like 
desktops,  integrate  not  only  productivity  tools,  such  as  Word,  Excel,  Access,  and  PowerPoint,  but  also  audio 
and  video  functions.  With  a microphone,  video  camera,  and  the  Internet,  a laptop  can  be  used  as  a voice  talking 
tool  and  a video  communication  tool.  Other  application  software  also  enhances  functions  of  a laptop.  Instant 
messaging,  like  MSN,  Yahoo  Messenger,  and  ICQ,  has  a value  of  presence  detection  and  has  become  a very 
popular  tool  for  text,  voice,  and  video  communications  in  both  one-to-one  and  group  formats  ( Wainhouse 
Research.  & First  Virtual  Communications,  Inc. , 2002).  Other  applications  that  may  facilitate  learning 
activities  and  interactions,  such  as  file  transfer,  stream  media,  whiteboard,  desktop  sharing,  and  tele -control,  can 
also  be  easily  installed  on  a laptop  computer.  The  trend  of  wireless  connection  further  fosters  the  growth  of 
“ubiquitous  computing”  on  campus  (Molenda  & Sullivan,  2002).  With  wireless  connected  and  sophisticated 
laptops,  digital  being  and  physical  being  are  merged  together.  Ubiquitous  computing  provides  new  educational 
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possibilities.  Mobile  learning  is  one  such  term  to  describe  a learning  format  via  portable  technology  (especially 
a laptop)  and  wireless  connection  in  the  context  of  ubiquitous  computing. 

While  technological  advances  have  made  mobile  learning  and  teaching  feasible,  cognitive,  pedagogical 
and  social  level  challenges  have  emerged.  Compared  to  other  research  in  the  field  of  instructional  technology, 
few  of  solid  research  studies  have  been  done  in  terms  of  mobile  learning.  Articles  related  to  laptop-using 
appear  more  online  than  in  journals  and  most  of  them  are  of  the  nature  of  evaluation  projects.  Through  these 
limited  research  studies,  we  can  still  learn  the  usage,  benefits,  and  barriers  regarding  laptop  computer-using, 
which  would  provide  us  a historical  perspective  and  assist  us  in  attacking  research  questions  for  this  study.  One 
of  the  most  high-profile  initiatives  is  Microsoft’s  Anytime  Anywhere  Learning  Program,  which  was  published 
as  the  Rockman  Report  (Belanger,  2002).  In  their  study,  Rockman  et  al.  (1998)  reported  significant  learning 
and  accomplishments  in  skill  development,  applications  of  technology  for  schoolwork,  and  improved  critical 
thinking  by  using  laptop  computers.  Rockman  et  al.(2000)  compared  two  groups  of  laptop-using  schools  and 
non-laptop-using  schools.  The  findings  indicate  that  laptop-using  students  have  more  technology  literacy  and 
skills,  better  performance  on  writing  assessment,  and  more  positive  attitude  toward  computers.  There  are  also 
many  other  research  studies  on  the  effect  of  students’  achievement  by  using  laptops  (Gardner  et.  al.,  1994; 
Stevenson,  1998;  Siegle  & Foster,  2001).  Gardner  et  al.  (1994)  got  mixed  findings  that  laptop-using  had  a 
positive  effect  on  science  achievement  while  no  effect  on  achievement  for  mathematics  and  English.  Gardner  et 
al.  also  report  that  students  with  laptop  computers  are  more  motivated  and  are  quicker  to  gain  technology 
literacy.  Steveson  (1998)  did  an  evaluation  with  focus  on  a standardized  achievement  test.  The  findings 
included  that  laptop  participants  significantly  outscored  their  non-participating  peers  on  test  and  laptop 
participants  have  better  school  attendance.  Siegle  and  Foster  (2001)  investigated  the  effect  of  the  using  of 
laptop  by  high  school  students  in  an  anatomy  and  physiology  course.  Two  groups  were  compared  by  using  a 
quasi-experimental  method.  The  findings  show  that  the  experimental  group  benefited  from  full-time  access  to 
laptop  computers,  has  overall  higher  achievement  and  increased  understanding  about  the  subject  matter.  Siegle 
and  Foster  also  indicated  that  laptop  computers  can  bridge  the  gap  of  the  access  to  technology  both  at  school 
and  at  home,  and  therefore  increase  opportunities  to  use  technology  as  an  authentic  integral  part  of  learning. 
Still  other  studies  (Carey  & Sale,  1997;  Hardy,  1999;  Healey,  1999)  discussed  laptop  using  in  terms  of  a larger 
social  context,  such  as  equity  issues,  and  educational  paradigm  change.  Carey  and  Sale  (1997)  reported  the 
benefits  of  using  laptops  for  students  with  severe  disability  in  cognitive  development.  Hardy  (1999)  discussed 
issues  of  equity  of  using  laptops  in  school.  Healey  (1999)  stated  that  laptop  using  wxild  benefit  student- 
centered  learning  and  facilitate  constructivism  paradigm. 

From  the  studies  discussed  above,  we  can  see  that  most  studies  of  using  laptop  computers  using  focus 
on  formal  school  situations.  Usually  laptop  computers  are  provided  by  schools,  corporations,  or  agencies, 
rather  than  by  students  themselves.  The  use  of  laptop  computers  usually  link  to  specific  subjects.  These 
subjects  include  English,  writing,  science,  mathematics,  geography,  social  studies,  etc.  Some  benefits  have 
been  recognized  in  those  studies: 

(1)  Improvement  in  students’  abilities  to  communicate  persuasively,  to  organize  ideas  effectively,  and 
to  use  vocabulary  effectively  (McMillan  & Honey,  1993). 

(2)  Laptop  using  improves  students’  technology  literacy,  and  enhances  students’  motivation  and 
independence  (Gardner,  Morrison,  Jarman,  Reilly,  & McNally,  1994). 

(3)  Benefit  for  students  with  disabilit  ies  (Carey  &Sale,  1997). 

(4)  Improves  students  ’ writing  skills  and  fosteis  collaboration;  allows  students  to  work  at  their  own 
pace  and  get  more  involved  in  schoolwork  (Rockman  et  al.,  1998;  Rockman  et  al.,  2000). 

(5)  Integrates  visual  presentation  with  written  notes  and  acquiring  information  with  processing 
information;  bridges  school  and  home  (Seigle  & Foster,  2001). 

As  mentioned  above,  most  studies  center  on  formal  learning  situations,  while  few  studies  focus  on 
informal  learning  environments.  Then,  why  is  an  informal  situation  important?  The  reason  is  that  the  situation 
of  laptop  using  has  changed.  Distributing  computers  by  schools  or  agencies  limits  the  sense  of  ownership,  even 
if  the  students  are  allowed  to  take  the  laptops  home.  Just  as  Siegle  and  Foster  (2001)  report  that  students  were 
concerned  about  damaging  the  computers,  laptops  in  those  studies  are  still  not  fully  accessible  and  exploitable. 
With  a limited  use,  a laptop  is  just  like  a desktop,  not  movable.  Learner  experience  is  still  not  totally  that  of 
movable  learning.  In  recent  years,  with  decrease  in  cost,  a laptop  has  become  a personal  tool.  Nowadays,  we 
can  see  quiet  a few  of  learners  bring  their  own  laptops  to  classrooms,  libraries,  or  conferences.  Different  from 
laptop  studies  mentioned  above  in  which  schools  or  corporations  distribute  laptop  computers  to  learners, 
learners  have  a total  ownership  of  laptops.  Total  ownership  makes  difference.  Laptop  computers  will  not  be 
used  as  an  experimental  variable  in  instruction.  No  teacher  present,  no  curriculum  requirement,  and  no 
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explicitly  measurable  objectives.  Learners  use  their  laptops  spontaneously  for  their  learning.  Just  like  most 
media  research  focuses  on  formal  situations,  research  regarding  laptop  using  also  falls  into  this  same  case. 
Research  on  the  role  of  laptops  in  the  informal  learning  environment  is  needed.  As  Salomon  et  al.  (1991) 
pointed  out  cognitive  effects  of  technology  using  are  reciprocally  interrelated  with  culturally  prescribed 
functions  and  modes  of  activities,  and  to  detect  technology  under  natural  conditions  may  reveal  us  what 
happens  otherwise  we  will  not  detect  under  experimental  context.  Thus  there  is  a gap  in  the  literature  of  mobile 
learners’  experience  with  laptops  in  an  informal  situation  or  in  other  words,  natural  conditions.  This  study  aims 
to  fill  such  a gap  with  the  focus  on  mobile  learners  in  higher  education  at  the  graduate  level. 

Research  Design  and  Methodology 

This  study  is  a qualitative  study  primarily  consisting  of  interviews  as  its  major  methods.  Three  semi- 
strutured  interviews  were  conducted  to  obtain  the  perspectives  of  adult  learners  in  higher  education  institutions 
who  were  using  laptop  computers.  Each  interview  was  conducted  individually,  face-to-face,  and  audio-taped. 


Context  and  Participants 

This  study  takes  place  in  a large  research-oriented  university.  Some  buildings  and  facilities  in  the 
university  have  begun  to  provide  wireless  connections  for  more  than  one  year,  which  are  free  to  public.  Due  to 
low  cost  and  easiness  to  build  WiFi  (W  ireless  Fidelity)  network,  more  and  more  wireless  access  zone  are  being 
built  around  the  downtown  next  to  the  university,  such  as  in  coffee  shops,  McDonalds,  and  other  entertainment 
areas.  “Going  mobile’’  is  becoming  a hot  topic  for  technology  pioneers  and  fans  in  the  university. 

Three  male  participants,  David,  in  his  20s,  Gary,  in  his  40s,  and  Sawn,  in  his  30s,  are  all  currently 
working  on  their  PhD  programs  in  the  college  of  education  in  the  university.  They  all  meet  participant  selection 
criteria  that  a participant  must  use  his  or  her  own  laptop  along  with  wireless  connection  for  more  than  half-year. 
Meanwhile,  they  all  value  and  enjoy  their  experience  of  using  laptop  for  their  study,  work  and  research. 

David  is  a fourth  year  doctoral  student  and  teaches  several  web  design  and  development  courses  for 
assistantship  in  his  department.  He  didn’t  like  laptop  very  much  until  he  bought  one  two  years  ago.  Now  he  is 
very  exciting  about  using  laptop,  especially  wireless  connection.  He  enjoys  doing  academic  working  in  the 
coffee  shop  and  see  there  as  his  office,  since  he  does  not  have  working  space  at  home  any  more  with  the  birth  of 
his  second  child.  He  believes  that  laptop  gives  him  much  convenience  and  saves  him  lots  of  time.  He  is 
concerned  that  the  screen  size  of  the  laptop  is  still  too  small  for  him  to  do  graphic  design  work. 

Sawn  is  a first  year  doctoral  student,  a previous  second  language  teacher,  and  has  owned  a laptop  for 
more  than  five  years.  His  initial  thought  to  buy  a laptop  was  to  learn  some  computer  applications,  like 
Photoshop  or  Excel.  However,  in  fact,  he  just  didn’t  learn  them  but  uses  laptop  mostly  for  writing,  literature 
searching,  and  doing  presentation.  He  loves  to  use  laptop  for  writing  because  he  believes  that  the  laptop 
computer  produces  better  documents  than  handwriting  does.  Although  he  uses  laptop  very  frequently,  he 
remains  skeptical  of  some  new  things  on  the  Internet,  like  Instant  Messaging,  Blog,  online  chatting.  He  almost 
never  uses  laptop  in  the  public  area,  except  in  classrooms.  In  addition,  he  still  wants  to  keep  formally  sitting  on 
a chair  and  the  laptop  on  the  desk.  He  thinks  he  grew  up  very  traditionally  and  he  still  likes  to  use  laptop 
traditionally. 

Gary  is  a second  year  doctoral  student  and  a lifelong  Georgia  resident.  He  loves  technologies  and  is 
always  ready  for  new  things.  He  has  owned  two  laptops  since  1998.  He  loves  to  learn  and  never  feels  trouble 
being  motivated  to  learn  new  things.  He  is  happy  that  the  laptop  provides  him  a third  office  so  that  he  does  not 
need  to  copy  files  between  his  home  office  and  school  office.  However,  he  agrees  that  it’s  a big  challenge  to 
get  back  to  work  when  the  Internet  always  take  him  away  from  his  original  work. 

In  sum,  researchers  attempt  to  focus  on  people  who  are  actively  involved  in  the  laptop  along  with 
wireless  connection  in  higher  education  institutions.  In  this  paper,  we  call  them  mobile  learner. 

Data  collection  process 

Semi -strutured  interviews  were  conducted  to  obtain  the  perspectives  of  mobiler  learneis  in  higher 
education  institutions.  The  interview  protocol  (see  Appendix),  which  focused  on  the  experiences  of  mobile 
learners  and  the  potential  impacts  on  cognitive  activity  by  using  laptop  computers,  was  used  to  guide  the  whole 
interview  process.  Researchers  identified  three  participants  through  daily  observation  in  the  college  of 

education.  The  criteria  to  choose  a participant  are  that  the  learner  must  be  a graduate  level  student  and  must 
own  a laptop  and  wireless  connection  for  more  than  half-year.  Interview  sites  were  in  the  building  where  the 
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college  of  education  located.  Two  interviews  took  place  in  the  interviewee’s  office,  and  one  interview  took 
place  in  a lounge  room  of  the  researcher’  department. 

Researchers  conducted  three  interviews  during  the  work  day  of  the  mid -spring  semester,  2004.  Each 
participant  was  expected  to  have  an  one-hour  face-to-face  interview,  which  would  be  audiotaped.  However, 
actually,  Sawn’s  interview  last  longer  than  one-hour,  and  Gary’s  interview  last  only  about  thirty-five  mintutes 
because  he  had  another  appointment  right  after  the  interview.  The  consent  form  was  distributed  to  the 
participants  before  the  interview  started.  The  participants  was  asked  to  read  and  signed  up  the  consent  forms. 

All  interviews  went  along  very  well  than  researchers  expected;  all  participants  seemed  very  active  to  express 
their  feelings  and  perspectives  about  laptops  and  mobile  learning.  It  was  the  limitation  of  the  record  tape  or 
time  (in  the  case  of  Gary)  that  stoped  the  conversation.  In  the  case  of  Sawn,  when  the  interview  was  over,  he 
continued  to  show  some  of  his  real  work  on  the  laptop.  He  demonstrated  how  to  create  a writing  project  by 
manipulating  his  previous  works  on  the  laptop.  So  besides  the  interview  data,  I also  collected  this  kind  of 
additional  observation  data. 

Data  analysis 

Audiotapes  collected  during  interviews  were  all  transcribed.  During  the  process  of  transcribing,  I 
already  begun  to  write  my  initial  thoughts  and  reflections  on  a separate  document.  I also  highlighted  some 
typical  senentens  and  keywords  on  the  transcripts  during  transcribing.  After  all  transcribing  tasks  are  done,  the 
transcripts  then  be  coded  by  low-level  labels  for  index,  and  typical  sentences  and  keywords  on  the  transcripts 
are  highlighted.  After  that,  the  codes  were  integated  for  each  participant  according  to  the  research  questions, 
and  then  were  input  into  a two-dimension  table.  One  deminsion  is  the  three  particiapants  and  another  is  the  four 
major  questions:  usage,  benefits,  limitations,  and  cognitve  impacts.  So,  that  is  3*4  table  as  an  approach  to 
reduce  data  and  seek  meaning.  Typical  sentences  and  keywords  were  also  put  into  each  related  cells.  The  two- 
deminsion  table  provides  a basis  for  comparison  among  cells . Sometime,  I adjusted  senetences  in  the  cells  and 
sometime  went  back  to  the  orignal  transcripts  again  for  double  check.  At  the  same  time,  I began  to  formalize 
the  themes  when  manipulating  the  table.  I initially  developed  fourteen  themes.  I went  back  to  the  transcripts 
and  questions  again  to  check  the  themes  and  make  revision.  For  some  themes,  especially  on  cognitive  effects,  I 
went  back  to  check  related  literature  s and  then  made  revision  on  the  themes. 

In  addition,  during  the  period  of  data  analysis,  I talked  informally  to  other  people  who  were  using 
laptop.  I talked  about  my  findings  and  asked  their  perspectives  on  using  laptop  in  higher  education  institutions. 
I also  attended  a mobile  learning  presentation  and  talked  with  people  who  were  showing  their  mobile  learning 
projects.  These  conversations  in  fact  also  helped  me  analyze  data  and  refine  themes. 

Research  Findings 

During  the  research  process,  we  realized  some  functions  participants  experienced  and  described  are  not 
necessary  confined  to  a laptop.  The  functions  like  cut  and  paste,  or  file  system,  or  the  Internet  access  are  also 
integrated  in  a desktop  computer.  However,  a laptop  makes  such  functions  more  personally  intimate  to  mobile 
learners,  which  help  form  a more  intimate  relationship  of  “intellectual  partneh'(Salomon  et  al.,  1991),  therefore, 
learners’  experience  with  those  functions  are  aggravated  in  a laptop  situation. 

Theme  1:  Laptop  serves  as  a “Mobile  Office’’  and  provides  possibilities  of  “Anywhere,  Anytime  Working’’. 

Participants  all  enjoy  freedom,  nobility,  and  portability  coming  with  laptops  and  wireless  connections. 
One  biggest  advantage  of  owning  a laptop  over  a desktop  is  that  a laptop  serves  as  a bridge  connecting  home 
and  office.  Gary  described  the  experience  before  and  after  owning  a laptop, 

“I  have  continuously  a problem  with  something  1 need  while  1 am  here  in  school  is  not  here,  it  is  in 
my  home  office.  But  something  I need  where  lam  working  in  my  home  office  is  not  with  me  because 
I left  it  in  my  school  office.  So  the  laptop  builds  a bridge  between  those  two.  ’’ 

“ Having  laptop  give  me  the  flexibility  to  work  at  my  office,  keep  my  work  take  home  conveniently 
work  at  home  and  come  back  again.  So,  it  is  much  more  efficient  arrangement  for  me  to  have  one 
laptop  which  is,  in  a way  the  laptop  is  my  third  office,  or  it’s  my  primary  office  maybe.  ” 

Moreover,  the  locations  of  using  laptop  are  already  beyond  office  and  home.  Such  locations  cover  library, 
classroom,  conference  site,  and  some  public  areas,  like  beaches.  With  a laptop,  adult  learners  can  move  among 
different  locations  freely.  David  described  such  freedom  and  convenience. 
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“ Having  everything  with  me,  I think,  no  matter  where  I am,  I can  work  on  the  couch.  I could  be  in 
my  office.  I could  go  to  coffee  shops.  Wherever  I am,  my  whole  office  is  with  me.  I have  all  my 
emails.  I have  all  my  notes  that  I have  written.  I have  all  my  papers  that  have  been  written  in  the 
past.  I have  all  the  website  that  I work  on.  Everything  is  there.  ” 

Meanwhile,  the  portability  of  a laptop  also  has  a benefit  of  accommodating  users’  different  preferences. 
Sawn  is  glad  that,  with  a laptop,  he  can  enjoy  working  at  home  now. 

“1  am  not  dependent  on  the  computer  lab  being  open.  I’m  not  dependent  on  facilities  being  open.” 

Interestingly,  participants  see  a laptop  more  as  a working  tool  than  a learning  tool.  Their  major  motivation 
to  purchase  a laptop  was  to  have  something  to  work  on.  Correspondingly,  they  describe  their  experiences  of 
using  laptop  more  as  “Anytime,  Anywhere  working”  than  “Anywhere,  Anytime  Learning”.  When  asked,  “how 
do  you  use  laptop  as  a learning  tool?”  Gray  showed  a perplex  about  how  to  link  laptop  to  learning  tool,  and  said 
“I  use  my  laptop  primarily  as  a productivity  tool.  It  does  help  me  to  generate  the  products  related 
learning,  like  writing  papers  and  creating  web  projects.  ” 

“I  think  it  is  more  as  of  a thinking  tool,  that  whatever  kind  of  task  maybe  involved  in  there’s  cdways 
some  way  that  a laptop  can  assist  in.  ” 

Main  tasks  performed  on  laptops  include  literature  locating,  paper  writing,  research,  teaching,  information 
access,  and  communications.  Maybe  because  they  are  graduate-level  students,  participants  seldom  use  laptop 
for  online  chatting  or  some  entertainment  activities.  This  may  be  a same  reason  why  the  participants  would  like 
to  describe  the  experience  of  using  laptop  in  terms  of  working.  The  participants  are  adult  learners  at  a graduate 
lever.  They  usually  have  multiple  roles,  as  a graduate  student,  a research  assistant,  a teaching  assistant,  and  a 
self-responsible  adult.  They  are  more  autonomous  in  the  decision  of  using  a laptop  as  a tool,  unlike  a K12 
student  who  will  use  a laptop  to  fulfill  teachers’  expectation.  Meanwhile,  their  learning  is  interwoven  with 
working,  rather  than  an  isolated  behavior.  They  prefer  to  learn  from  working  or  doing  in  some  real  tasks.  The 
vocabulary  of  working  definitely  matches  more  their  roles  and  autonomy. 

Theme  2:  laptop  provides  learners  a “shortcut”  and  allows  learners  to  have  instant  access  and  instant  action. 

Laptop  not  only  provides  connection  between  physical  locations,  but  also  provides  a connection  between 
thoughts  and  action,  which  here  I referred  to  as  a “shortcut”.  All  participants  reported  that  they  use  the  Internet 
and  World  Wide  Web  to  find  information  they  need.  When  an  adult  learner  comes  up  with  a question,  laptop 
provides  a shortcut  to  gain  information  through  the  network.  It  also  makes  instant  access  possible  in  the 
classroom  activity.  Laptop  serves  as  a short  cut  between  questions  and  answers  when  the  topic  involves  online 
resources . Gary  gave  an  example  in  which  using  laptop  for  the  Internet  access  is  a part  of  the  classroom 
discussion  and  conversation. 

“We  can  verify  answers  to  some  questions,  or  check  some  schedules,  or  check  some  information 
onlin  e.  ” 

The  shortcut  may  also  refer  to  as  information  access  on  learner’  own  laptop.  Gary  described  the  advantage 
of  using  endnotes  for  references, 

“Once  records  is  there,  it’s  wonderful  to  be  able  to  access  very  quickly  when  I work  on  the  new 
document  for  citation  purpose.  ” 

The  shortcut  implies  instant  actions  as  well.  When  an  idea  comes  up,  the  participant  usually  will  set  up  a 
framework  of  what  is  going  to  be  done  in  a very  short  time  and  then  start  writing  or  creating  a project  more 
quickly.  Sawn  illustrated  his  writing  process, 

“ Because  of  the  storage  system  on  the  laptop  and  I was  able  to  cut  and  paste  parts  of  it.  And  so  1 
was  already  half  ways  done.  ” 

Instant  access  implies  that  learner’s  behaviors  and  classroom  activities  are  changing.  This  may  also 
provide  feasibility  of  new  instructional  models,  say,  resources  -based  learning.  However,  instant  access  may 
also  have  some  negative  influences.  Gary  said, 

“My  habit  that  actually  go  to  the  main  library  or  science  library  to  access  material  there,  that  habit 
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has  suffered” 

Theme  3:  Laptop  facilitates  three-dimension  of  writing,  a metaphor  of  clays  and  sculpture. 

One  major  task  on  laptop  is  writing.  Writing  on  the  laptop  is  changing  the  composing  process.  A vivid 
metaphor  for  the  writing  process  on  the  laptop  can  be  seen  as  a sculptor  manipulating  the  clays  to  make  a 
sculpture . The  change  of  writing  process  results  from  at  least  three  factors . First,  because  of  the  function  of 
“cut  and  paste”,  the  writing  process  is  no  longer  linear,  and  therefore,  thinking  during  writing  process  is  less 
linear.  Sawn  described, 

“It  becomes  more  of  organics,  piece  that  you  can  manipulate  in  ways  that  you  can ’t  with  a pen  on 
paper,  for  definitely  ” 

“I  love  cut  and  paste  because  it  is  just  like,  why  not  change  everything  because  you  have  the  old 
one.  ” 

Second,  the  previous  works  stored  in  the  laptop  make  writers  to  create  a new  document  faster,  and 
therefore,  proportion  of  writing  is  more  about  revision. 

“ Because  of  the  storage  system  on  the  laptop  and  I was  able  to  cut  and  paste  parts  of  it.  And  so  1 
was  already  half  ways  done.  And  then  what  I found  too  was  the  revision,  let's  see,  I was  thinking 
about  revising  this  piece,  or  making  it  to  something  different.  ” 

“I  used  some  pieces  to  start  a new  piece  or  to  start  a new  project,  and  it  is  just,  I fount  it,  I found  it 
to  be  real  useful.  ” 

Third,  the  editing  and  revisbn  process  happen  simultaneously.  Learners  enter  the  whole  writing  process 
quicker.  As  Sawn  stated, 

“When  I began  writing  something,  I already  revise  it  in  a very  visional  way  and  in  a very,  and 
editing  it  and  start  spell  check  on  grammar  check,  already  begin  on  revising  and  editing,  and 
where 's  handwritten  thing  you  don ’t  do  that,  no.  ” 

Therefore,  mobile  learners  are  experiencing  inch  freedom  in  writing.  The  paper-based  linear  writing 
process  is  changed,  and  the  composition  process  of  writing  a paper  is  more  like  manipulating  some  pieces.  This 
nonlinear  writing  process  may  favor  students  who  are  not  good  at  traditional  writing. 

Theme  4:  Search  and  organization  function  of  a laptop  is  a plus  of  brain  retrieval  system. 

Search  and  organization  function  is  one  of  mobile  learners’  most  favorite  functions.  The  function  can  also 
be  found  in  the  desktop,  however,  laptop  users  keep  everything  on  the  laptop  and  therefore,  the  function  seems 
more  beneficial  for  them.  The  search  and  organization  function  definitely  shares  the  cognitive  load  for  users 
who  are  in  need  of  managing  materials  and  projects.  Sawn  is  very  happy  about  this  function, 

“The  laptop  itself  simplified  tremendously.  In  terms  of  storage,  in  terms  of  finding  what  you  wrote 
or  piece  that  you  did  even  when  it  was  five  years  ago.” 

Gary  shares  very  similar  feeling, 

“I’m  looking  for  a document  on  my  computer  something  I created  three  months  ago  or  some 
resources,  the  search  function  in  my  computer  is  the  obvious  way  to  help  me.” 

Such  function  is  not  just  for  searching  and  organizing  the  word  documents,  but  also  includes  all  other 
materials  on  the  laptop,  like  PDF  or  HTML  files.  It  serves  as  a whole  resource  center  or  system  for  the  mobile 
learners  who  have  begun  to  give  up  using  paper-based  medium  to  store  and  organize  their  academic  works. 
David  described  his  experience  in  this  way. 

“I  figure  out  a good  system  that  I could  have.  I could  look  up  the  endnotes,  and  then  I get  number, 
the  catalog  number  I get  it,  and  I know  if  I have  it  printed  it  out  in  my  cabinet.  Where  it  is?  If  I 
have  a PDF  version,  where  it  is?  ” 

“And  I index  any  notes  that  I put  in.  I index  all  my  research  notes.  Any  reading  notes  I take,  I index. 
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All  my  papers  get  indexed.  ” 

Gary  gives  an  example  of  using  DreamWeaver  as  an  organizing  tool. 

“ The  tool  like  DreamWeaver  gives  you  that  automatic  organizing  of  the  hierarchical  relationship  of 
those  documents  and  the  hyperlink  kind  of  relationship  with  the  documents.  That  is  lots  of  easier  for 
me  to  conceptualize  with  the  program’s  help,  then  for  me  just  to  deal  with,  do  way  that  designer  just 
has  to  do  it,  which  is  just  folder  and  documents,  and  then  keep  tracking  of  everything  in  your  brain.” 

Certainly,  participants  are  satisfied  with  using  laptop  to  organize  the  materials  and  projects.  They  feel  like, 
“It  surely  makes  me  more  organized  researcher.  ” 

“That’s  reason  why  I like  using  notebook  so  much,  it  sen’es  to  develop  my  system  from  the  ground 
up  with  the  notebook  as  a factor,  I didn ’t  translate  from  paper  based  system.  ” 

Computer-based  searching  and  organizing  system  simplifies  the  management  and  retrieval  of  materials.  It 
serves  as  an  extension  of  mind  by  sharing  lower-level  workload  so  that  learners  can  focus  more  on  meta-level 
management,  design  level  activity,  or  higher- level  activity. 

Theme  5:  No  game,  no  entertainment,  but  participants  still  feel  trapped  by  the  Internet  and  email. 

Those  mobile  learners  all  use  their  laptops  for  academic  purposes.  They  all  reported  that  they  rarely  used 
laptops  for  entertainment.  However,  they  feel  easy  to  get  lost  in  the  cyberspace.  It  has  been  a consistent 
problem  for  learners  who  have  the  Internet  connection.  Gary  described  such  trouble, 

“7  think,  overall,  it  still  help  me  more  productive;  however,  the  laptop  in  front  of  me  connected  to 
the  Internet  is  always  temptation  to  other  kind  of  distraction.  The  World  Wide  Web  is  there,  waiting 
for  you,  so  there  is  a little  person  concerns  with  this.  ” 

“ During  any  given  day,  I am  spending  too  much  time,  just  checking  news,  checking  the  weather, 
checking  email,  umm,  and  you  know,  kind  of  browsing,  and  that  kind  of  thing,  instead  of  intending  to 
my  next  task.  ” 

David  also  agreed  that  the  Internet  was  so  distractive  and  expressed  the  same  feeling  that  the  Internet  eats 
up  lots  of  time, 

“It’s  great  to  have  wireless,  though.  If  you  really  need  to  write,  not  having  the  connection  is  nice, 
sometimes.  ” 

If  temptation  of  the  Internet  access  happens  in  classrooms,  then  it  introduces  the  off-task  behavior.  David 
described  his  feeling  when  using  laptop  for  note  taking  in  the  classroom, 

“When  it  (laptop)  is  on  in  front  of  me,  I am  looking  at  it,  you  know,  I have  a hard  time  to  look 
away.  ” 

Of  course,  such  consistent  problem  caused  by  the  Internet  does  not  mean  that  school  should  cut  off  the 
Internet  connection.  However,  in  the  mobile  learning  pedagogy,  we  do  need  to  address  this  issue  to  avoid  too 
much  off-task  behavior. 

Theme  6:  Using  laptop  for  note  taking  in  the  classroom  still  presents  two  folders  for  mobile  learners. 

Currently,  quiet  a few  of  students  begin  to  bring  their  laptops  to  classrooms.  They  mainly  use  it  for  note 
taking.  Learners  prefer  to  use  a laptop  as  a note-taking  tool  for  several  reasons.  One  reason  is  that  the  laptop 
generates  better  notes  than  handwriting,  and  another  reason  is  that  computer-based  notes  are  easier  to  manage. 
Sawn  said, 

“7  don’t  have  to  write  it  by  hand  and  then  copy  into  computer.  I can  do  it  in  computer.  It  cuts  the 
time,  and  then  I found  the  notes  were  a much  better.  ” 

“Actually,  I think  it  was  really  a great  idea,  because  what  I used  to  do  is  1 take  notes  on  paper  and 
then  I lose  the  paper.  Or  I have  to  carry  all  that  notebooks  to  class  and  it  was  too  heavy.  ” 

However,  not  all  participants  feel  totally  comfortable  using  laptop  for  note  taking  in  classrooms.  There  are 
some  psychological  factors  existed,  such  as  be  afraid  of  being  thought  as  impolite  or  as  doing  some  other 
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businesses.  Gary  described  such  factors, 

“ People  notice  you  have  it,  if  you  type,  then  there  is  some  sounds,  although  mine  is  very  quiet,  but 
still  is  not  silent,  so  when  I type,  there ’s  still  some  noise  it  makes.  It  also  the  screen,  the  laptop 
screen  tends  to  block  my  view  a little  bit  of  instructor  or  other  people  in  the  room.  ” 

“I  am  rather  sensitive  to,  I don ’t  want  to  be  impolite  or  I don  ’t  want  to  be  disrespectful  to  my 
professor  or  my  classmates.  So,  I don ’t  let  myself  do  very  much.” 

In  addition,  laptop  itself  does  introduce  off-task  activities,  at  least,  for  some  learners.  David  described, 

“It’s  certainly  like  putting  a TV  in  front  of  me.  I have  a hard  of  time  to  pay  attention  to  the 
instructor.  I want  to  be  watching  TV.  And  that’s  same  thing  with  computer.  When  it  is  on  in  front  of 
me,  I am  looking  at  it.  I have  a hard  time  to  look  away.  ” 

Theme  7:  Not  being  able  to  see  the  whole  thing  and  losing  a certain  level  of  interaction  are  two  major  concerns 
for  those  mobile  learners. 

Besides  many  other  concerns  like  lasting  period  of  battery  or  computer  virus,  the  two  concerns  mentioned 
in  the  subtitle  seem  more  related  to  inner  mental  functioning.  Those  participants  all  reported  that  they  would 
like  to  have  printed  documents  in  some  cases,  especially  when  they  want  to  have  a whole  picture  of  a larger 
document  or  want  to  make  comments  in  the  document.  Printed  documents  provide  some  interactions  that 
laptop  can  not  provided.  Sawn  feels  that  it  is  necessary  to  see  the  whole  document  in  the  writing  process.  He 
said, 

“Sometime  I like  to  see  whole  document  at  once,  like  spread  out  on  the  table,  where  is  laptop  still 
doesn ’t  allow  me  to  do  that.  ” 

“That’s  a limitation,  which  is  why  I still  like  to  print  thing  out  when  l am  on  drafting,  when  I feel  I 
have  a good  draft,  because  I want  to  look  at  the  whole  piece.  I want  to  be  able  to  page  and  hold  one 
page  and  look  at  the  other  page.  ” 

“Periodically,  after  been  working  on  it  for  two  or  three  days,  I’ll  print  out  it  and  take  a look, 
because  I’ll  see  thitigs  that  1 won’t  find  on  the  laptop.  So  maybe  they  compliment  each  other,  I 
guess.  ” 

David  described  the  difficulty  of  putting  comments  on  computer  documents. 

“It’s  a good  feel  (to  hold  a printed  document)  on  your  hand.  Hold  the  marker.  Underline  and 
make  a little  note.  You  lose  that  ability  on  the  computer.  ” 

“I  can  put  notes  into  Adobe  Acrobat  file,  and  I could  circle  something  in  it,  but  I got  to  the  menu, 
click  draw  tool,  and  circle  it,  put  note  in.  You  just  could  not  do  it.  I keep  on  grading  my  students 
assignments  on  the  computer,  and  nine  out  of  ten,  Td  print  them  out,  because  it’s  just  so  much 
faster,  just  mark,  mark,  mark,  mark,  A minus,  you  know,  rather  than  focus  on  how  are  you  going  to 
put  comments  in.  ” 

Definitely,  these  two  concerns  come  with  the  technology  and  also  depend  on  the  advance  of  technology. 
They  also  reflect  the  needs  for  cognitive  functioning,  which  have  to  be  addressed  in  technology-based  learning 
environments . 

We  have  described  several  themes  emerged  from  mobile  learners’  experience,  behavior,  and  perception. 
As  a summary,  more  details  on  mobile  learners’  experience  are  presented  in  Table  1.  These  experiences  are 
usually  interwoven.  Take  the  metaphor  of  mobile  office  as  an  example.  It  can  either  be  seen  as  a usage,  an 
effect  or  a benefit  of  using  laptop.  On  the  other  hand,  however,  it  may  be  a byproduct  to  learners  with  a 
concern  of  “anywhere,  anytime  working’’. 
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Table  1.  The  experience  of  mobile  learners 


Behavior,  Outcomes,  and  Patterns  regarding  Laptop  Using 

Usage 

Write  papers,  e.g.  spell  check,  and  grammar  check,  cut  and  paste 
Do  research,  e.g.  field  notes  for  observation;  grand  applications 
Do  assistant  work,  e.g.  lesson  planning 
Search  information  and  access  the  resources  on  the  Internet 
Store  and  organize  materials,  e.g.  keeping  notes,  using  Endnotes 
Use  for  communications,  e.g.  email,  presentation 
Support  team  working,  e.g.  using  track  change  and  comments 
Take  notes  in  class 

Prepare  all  class  work,  including  comprehensive  exam 
Create  web  projects 
Do  graphic  design 

Effects  with 

Better  quality  and  format  of  documents 
Less  repetitive  works 
Replace  for  handwriting  and  paperwork 
A coherence  form  to  organized  files 
Shortcut  between  thoughts  and  action. 
Mobile  office 

Effects  of 

Three  Dimension  of  writing:  manipulating  the  writing  pieces 

Accommodation  for  personal  preference  and  habit 

Be  a productive  constructer  of  the  knowledge 

More  like  a designer,  rather  than  a learner  in  front  of  a laptop 

Motivated  to  be  organized 

Intimate  partner  and  tool 

Limitations 

Battery  dies;  (Mobile  learners  like  to  have  power  outlets  in  rooms ) 
Temptation  from  the  Internet 
Email  eats  up  time 

Note  taking  in  class  is  still  not  comfortable 

Reading  on  the  laptop  loses  a certain  level  of  interaction  compared  to 
reading  on  the  printed  materials 
Not  being  able  to  see  the  whole  thing 

Identity 

Productivity  tool,  thinking  tool,  extension  of  capability  and  the  brain. 

Final  Words 

Because  of  its  portability  and  mobility,  a laptop  computer  has  become  a personally  intimate  partner  for 
a learner.  Impacts  of  mobile  technologies  have  many  direct  effects,  like  better  quality  and  format  of  documents, 
or  less  repetitive  works,  as  well  as  many  latent  effects,  like  promoting  collaboration  among  peers  or  increasing 
motivation  of  a learner  to  be  more  organized.  Such  an  intimate  relationship  is  two  folds.  It  provides 
convenience  and  instant  access  for  learners.  However,  it  may  also  eat  up  time  of  learners  or  become  a 
distracting  factor  during  the  learning  process.  Partnership  may  be  a most  significant  characteristic  of  mobile 
learning  environments  comparing  to  other  technology  based  learning  environments.  Educational  computing  has 
undergone  a change  of  focus  from  e-learning  to  mobile  learning.  More  research  studies  should  be  done 
regarding  how  psychologies  factors  are  influenced  by  this  characteristic  of  partnership.  Future  research  could 
also  look  into  the  relationship  of  mobile  technologies  and  learners  in  a broader  mobile  learning  space. 
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Abstract 

This  paper  expands  on  a poster  presentation  about  the  results  of  research  in  PBL  in  distance  education 
environments  integrated  in  the  PBL  design  model  as  defined  by  traditional  literature.  The  results  will  be  laid 
out  as  they  are  relevant  to  each  stage  of  the  process.  Research  on  discrete  aspects  of  its  design  process  is 
necessary  to  enlighten  understanding  of  the  particular  areas  that  need  to  be  modified  when  PBL  is  implemented 
over  distance  education.  Nevertheless,  these  results  need  to  be  consolidated  into  design  models  that  will  guide 
designers  in  their  pursuit  of  creating  online  experiences  that  benefit  from  constructivist  principles  of  learning.  A 
related  goal  is  to  make  evident  current  strands  of  research  and  highlight  possible  directions  for  continued 
research  that  enlightens  instructional  designers  as  they  translate  the  methodological  specifications  of  PBL  into 
DE  environments.  Continued  study  and  incorporation  of  findings  in  this  way  can  even  provide  evidence  for  the 
continued  adoption  of  traditional  PBL  models  for  the  DE  design  or  the  need  of  departure  from  them  to  create  a 
specialized  PBL  for  DE  model. 


Introduction 

PBL  has  been  characterized  as  an  example  of  constructivist  thinking  (Duffy  & Cunningham,  1996). 
The  design  principle  of  rooting  educational  activities  in  a realistic  situation  that  embodies  the  knowledge  and 
skills  that  students  need  to  learn  is  one  of  the  ways  in  which  it  implements  constructivist  principles.  The  use  of 
this  kind  of  authentic  situation  stimulates  students  to  go  beyond  memorizing  facts.  It  encourages  them  to 
analyze  and  evaluate  facts  and  skills  and  integrate  them  into  a cohesive  approach  to  transform  the  current  state 
of  the  situation  into  an  ideal  one  according  to  the  circumstances.  Students  are  transformed  into  active  role 
players  with  responsibilities  that  are  similar  to  the  ones  they  would  assume  in  the  world  of  practice.  The  pursuit 
of  knowledge  now  becomes  a dynamic  undertaking  that  motivates  students  because  of  the  immediacy  and 
evident  relevance  of  its  application.  Consistent  with  its  goal  of  making  learning  a mirror  of  practice,  most 
implementations  encourage  collaboration  and  communication  among  learners  during  this  process.  Learners 
bring  together  their  previous  experiences,  skills  and  current  understandings  into  a negotiation  process  that 
should  contribute  to  strengthen  students’  response  to  the  learning  situation.  The  process  of  consultation  of 
sources  of  information,  negotiation  of  understandings  and  design  and  refinement  of  the  solution  is  backed  up  by 
continuous  reflection  on  content  and  process.  It  is  improved  on  an  ongoing  basis  by  input  from  multiple  sources 
and  perspectives  of  assessment.  The  result  constitutes  an  informed  response  of  students  to  address  the  situation. 

Distance  education  is  becoming  a realistic  alternative  for  many  to  further  their  academic  careers.  The 
benefits  that  authors  have  attributed  to  distance  learning  have  spanned  at  least  financial,  accessibility  and 
academic  reasons  (Belanger  & Jordan,  2000).  One  of  the  long-standing  concerns  with  it  is  whether  the  quality 
of  education  parallels  that  of  traditional  classroom-based  education  (Bimbaum,  2001).  An  approach  to  try  to 
ensure  this  is  to  design  educational  experiences  that  benefit  from  the  educational  principles  that  are  deemed  to 
yield  effective  learning.  PBL  is  one  of  the  educational  methodologies  that  have  emerged  from  practice  in 
traditional  classroom-based  environments  as  a successful  approach  to  strengthen  learning.  Therefore,  designing 
distance  learning  experiences  with  PBL  can  be  one  of  the  alternatives  by  which  institutions  provide  equal 
quality  distance  education  to  their  student  populations. 

Different  strands  in  communication  theory  advocate  the  preponderance  of  media  in  the  delivery  of 
messages  (Meyrowitz,  1994).  Authors  like  Marshall  McLuhan  are  cited  for  his  ideas  of  the  influence  media  has 
in  the  delivery  of  messages  (Deibert,  1998).  Other  authors  stress  the  inherent  importance  of  physical  immediacy 
in  human  interaction  (Boden  & Molotch,  1994).  In  light  of  the  ideas  of  these  currents  of  thinking,  the  intention 
of  fostering  collaborative  problem-based  learning  environments  at  a distance  needs  the  special  consideration  of 
the  affordances  and  challenges  that  the  medium  of  delivery  attaches  to  it.  Synchronous  communication  that 
could  bring  the  benefits  of  copresence  is  technology-mediated.  Asynchronous  communication  that  brings  the 
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advantage  of  reflective  thinking  entails  delays  in  the  exchange  of  ideas.  These  situations  suggest  that 
implementation  of  such  distributed  examples  of  problem-based  learning  need  to  be  informed  by  research  that 
considers  the  new  factors  that  come  to  play  in  it. 

One  of  the  common  denominators  in  problem-based  learning  models  is  the  use  of  collaborative 
learning  groups  during  the  process.  Cooperation  in  learning  has  been  defined  as  “working  together  to 
accomplish  shared  goals”  (Johnson  & Johnson,  1996).  These  authors  stress  the  importance  of  various  aspects 
that  characterize  group  work.  The  first  aspect  they  mention  is  positive  interdependence.  It  has  to  do  with  the 
idea  of  working  towards  the  accomplishment  of  a shared  goal.  They  also  talk  about  the  importance  of  both 
individual  and  group  accountability.  Another  important  aspect  is  the  development  of  interpersonal  skills.  Yet 
another  aspect  is  the  ability  to  self  monitor  group  work  to  ensure  consistent  progress  towards  the  goal  and  to 
discontinue  patterns  of  behavior  that  impede  this  progress.  However,  the  last  item  that  these  authors  mention  is 
what  constitutes  one  of  the  challenges  that  this  kind  of  inquiry  wishes  to  explore.  They  explicitly  incorporate 
face-to  face  interaction  as  one  of  the  defining  aspects  of  group  work.  When  learning  necessarily  has  to  occur 
with  a separation  of  space  and/or  time,  educators  then  need  to  know  how  to  facilitate  such  experience  with  the 
lack  of  immediacy  between  distance  learners  and  teachers. 

The  Literature  Review 

Research  and  evaluation  literature  on  the  development  of  PBL  in  DE  has  already  begun.  Researchers 
and  practitioners  have  started  to  help  identify  possible  limitations  and  / or  advantages  of  this  combination.  This 
section  describes  the  findings  of  several  research  undertakings  as  they  zoom  into  general  or  more  discrete 
relevant  portions  of  the  PBL  design  and  implementation  process. 

Teachers  and  Students 

Sage  (2000)  studied  the  overall  PBL  experience  from  the  perspective  of  students  and  instructors 
involved  in  the  process.  She  focused  on  a set  of  elements  each  with  a continuum  of  possible  values,  the 
combination  of  which  has  an  impact  on  the  online  PBL  experience.  Courses  that  attempt  to  implement  online 
PBL  will  encounter  several  starting  characteristics  of  teachers  and  students  that  will  be  somewhat  given.  These 
cannot  necessarily  be  changed  during  a single  learning  experience.  Teachers  and  students  bring  their 
assumptions,  skills  and  preferences  related  to  both  teaching  and  learning.  They  also  bring  their  previous 
experiences  and  abilities  in  teaching  and  learning  in  traditional,  constructivist,  PBL  and  online  environments. 
The  more  experienced  both  teachers  and  students  are  in  all  the  factors  that  compose  the  PBL  experience,  the 
more  flexibility  educators  will  have  to  implement  experiences  that  are  closer  to  the  models  of  this  methodology 
that  view  students  as  more  self-directed. 

Taplin  (2000)  reports  on  the  experiences  of  educators  who  are  beginners  in  the  transition  from  more 
traditional  educational  methods  to  the  implementation  of  online  PBL.  She  also  points  out  the  importance  of 
considering  student  characteristics  in  the  design  of  the  course  particularly  regarding  their  flexibility  to  devote 
time  to  identifying  and  evaluating  resources  by  themselves,  individual  accountability  and  group  work.  The 
limited  schedules  of  distance  learners  is  what  makes  them  turn  to  anytime  anywhere  flexible  opportunities  for 
learning.  Their  availability  needs  to  be  taken  into  account  and  balanced  with  provision  of  resources  and  the 
design  of  group  experiences  so  the  assumed  highlights  of  such  undertakings  do  not  turn  into  deterrents  of 
learning.  Teacher  experience  and  availability  to  facilitate  is  also  deemed  important  by  this  author. 

Poon  (1997)  describe  a hybrid  environment  in  which  educational  efforts  are  triggered  by  problems  that 
depict  what  students  can  do  within  a subject  domain  instead  of  what  students  should  know.  The  distance 
learning  technologies  together  with  face-to-face  experiences  help  deliver  the  content  that  students  will  use  in 
order  to  solve  the  problems.  The  face-to-face  component  is  also  the  setting  in  which  students  encounter  the 
problem  and  initialize  the  process  of  problem  definition  and  process  organization.  Then  students  undergo  the 
iterative  process  of  consulting  sources  of  information  and  devising  the  solution.  In  the  final  stage,  students  not 
only  construct  the  solution,  but  also  reflect  about  what  they  have  done  and  relate  it  to  future  practice. 

These  authors  focused  on  feedback  from  tutors  at  the  end  of  the  first  stage  of  work.  This  feedback 
covered  areas  such  as  ability  of  tutors  to  discern  the  scope  of  content  they  needed  to  care  for  based  on  the 
problem,  understanding  of  the  nature  and  purpose  of  PBL,  the  need  of  both  training  and  practice  to  internalize 
the  approach,  time  demands  imposed  by  new  teaching  skills,  reinforcement  of  change  in  student  roles  (from 
passive  to  active),  shift  in  the  role  of  teachers  (from  providing  knowledge  to  questioning,  making  resources 
available,  and  refocusing)  and  problem  generation.  This  experience  underscores  the  need  for  teacher 
development  when  attempting  to  implement  PBL  designs.  It  also  highlights  the  usefulness  of  considering 
teacher  feedback  to  improve  the  design  of  such  environments. 
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Readiness  to  Work  in  a PBL  Environment 

Readiness  to  work  in  a PBL  environment  not  only  benefits  performance  in  learning  but  also  in 
professional  practice  (Bjorc  k,  2002).  His  study  focused  on  the  nature  of  interactions  as  a way  to  evidence  the 
level  of  appropriation  and  mastery  of  the  process.  The  researcher  studied  issues  of  participation:  amount  and 
content  of  messages.  Findings  suggested  a relationship  between  the  volume  and  nature  of  interaction  and  the 
level  of  mastery  of  online  PBL.  Mastery  was  signaled  by  richer  communication,  more  and  detailed,  critical  but 
respectful  questioning  of  classmates’  standpoints,  openness  to  feedback,  continuous  and  spontaneous 
discussion.  A shift  in  the  facilitator’s  role  with  progressively  less  intervention  evidenced  students’  level  of 
confidence  in  their  skills  to  undergo  the  process. 

This  author  concludes  that  multiple  online  behaviors  exhibited  by  students  describe  their  level  of 
mastery  of  online  PBL.  Nevertheless,  for  students  to  display  the  highest  level  of  those  characteristics,  it  takes 
several  iterations  of  participating  in  the  process.  Initial  scaffolding  may  be  crucial  in  whether  mastery  of  online 
PBL  may  be  accomplished  among  students  but  eventual  fading  will  characterize  their  achievement  of  it.  The 
findings  of  this  author  seem  to  indicate  the  need  of  finding  ways  for  strengthening  student  mastery  of  the 
process  in  the  PBL  design  because  of  the  dual  benefit  for  the  process  itself  and  for  future  professional 
performance. 

Institutional  Arrangement  and  Support 

Sage  (2000)  also  explored  the  influence  of  the  virtual  structural  environment  of  the  course.  The 
structural  environment  suggests  that  the  course  will  have  a previously  specified  amount  of  students  and  length 
and  will  be  backed  up  by  certain  types  of  support  from  the  institution.  Therefore,  educators  need  to  adjust  those 
PBL  experiences  based  on  the  combined  constraints  and  / or  particular  advantages  that  these  elements  will 
afford.  Taplin  (2000)  had  similar  findings  in  that  the  overall  commitment  and  support  of  the  relevant 
administrative  and  academic  departments  is  seen  as  crucial  for  the  successful  online  PBL  experience. 

Subject  Matter 

The  nature  of  the  subject  matter  will  also  impact  the  design  of  the  experience  (Sage,  2000).  The 
amount  and  level  of  complexity  of  the  information  that  students  are  expected  to  learn  helps  to  define  the  kinds 
of  problems  that  can  be  adopted  as  starting  points  for  learning.  Then  the  characteristics  of  the  problem  (e.g. 
level  of  definition)  and  the  amount  of  resources  provided  by  the  instructor  also  have  an  impact  in  design 
decisions. 

Technology  Literacy  and  Infrastructure 

The  technology  access  and  support  that  the  course  receives  is  yet  the  last  aspect  that  Sage  (2000) 
describes.  The  type  of  communication  and  collaboration  tool  that  the  course  uses  together  with  its  advantages 
and  disadvantages  for  supporting  PBL  contributes  to  strengthen  or  debilitate  the  PBL  implementation. 

McAlpine  & Dudley  (2001)  studied  an  implementation  of  online  PBL  in  a course  that  consisted  of  5 
PBL  experiences  two  to  three  weeks  long  each.  It  used  online  communication  to  support  the  exchange  of  ideas 
towards  the  resolution  of  the  problem.  They  found  that  both  students  and  the  institution  lacked  the  appropriate 
technology  infrastructure  to  support  the  process.  Technology  literacy,  access  and  support  can  become  important 
obstacles  in  the  development  of  effective  online  PBL.  Technology  requirements  are  at  the  threshold  of  an 
educational  experience:  it  is  a supporting  element  that  must  be  transparent  to  all  users  so  it  will  not  severely 
limit  learning  benefits.  This  is  consistent  with  Sage’s  (2000)  findings. 

Impact  of  the  PBL  Problem  in  Interaction 

The  connection  between  the  type  of  problem  that  underlies  a PBL  experience  and  the  amount  and 
nature  of  interactions  that  occur  during  its  process  is  the  area  of  interest  of  Ronteltap  & Eurelings  (2002). 
Learning  issues  identified  by  students  based  on  the  problem  were  classified  as  theoretical  or  practical.  The 
authors  analyzed  quantitatively  the  amount  of  interactions  generated  by  type  of  issues.  They  also  focused  on 
their  level  of  cognitive  activity  (e.g.  low  for  copy  and  paste  and  high  for  original  contributions).  The  study 
relied  on  student  and  instructor  interviews  and  analysis  of  discussion  board  transcripts  as  multiple  sources  of 
information  to  corroborate  findings. 

According  to  this  study,  practical  learning  issues  increased  the  amount  and  quality  of  interaction 
between  students.  The  need  for  continued  research  on  this  is  established  due  to  a small  sample.  Nevertheless, 
these  results  could  be  supporting  the  view  that  learning  that  consists  of  memorization  of  facts  only  yields 
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enough  interaction  to  achieve  their  reproduction.  Furthermore,  educational  experiences  that  aim  at  developing 
increased  scholarly  interaction  in  quality  and  quantity  should  utilize  practical  problems  as  their  starting  point. 
This  study  provides  useful  pointers  for  educators  looking  for  ways  to  increase  interaction  especially  to  avoid  the 
sense  of  isolation  that  some  distance  learners  experience. 

Group  Development 

McConnell  (2002)  utilizes  analysis  of  online  discussions,  products  and  student  interviews  to  examine 
the  development  of  PBL  groups  throughout  the  process.  The  course  guides  students  to  learn  the  design  and 
evaluation  of  learning  that  occurs  through  PBL  (which  constitutes  experiential  learning).  They  are  given 
ownership  in  determining  the  focus  of  the  problem,  direction  of  their  efforts,  monitoring  their  process,  and 
simultaneously  evaluating  and  redesigning  the  process  in  which  they  are  participating.  The  facilitator  becomes  a 
co-participant  in  the  process  relinquishing  most  of  its  traditional  authority  in  making  decisions. 

The  author’s  findings  point  to  the  development  of  groups  in  three  consecutive  but  overlapping  stages. 
The  first  stage  was  characterized  by  negotiation  of  understandings  and  organization  decisions.  The  second  stage 
comprised  the  research  related  to  devising  the  solution.  The  third  stage  corresponds  to  the  development  of  the 
final  product.  This  author  found  iterations  within  the  first  stage  and  simultaneous  interaction  among  members  of 
the  group  and  its  subgroups.  This  author  describes  implications  of  his  research  for  the  design  of  online  PBL. 
Because  of  the  affordances  of  the  online  environment,  the  development  of  groups’  stages  occurs  simultaneously 
as  the  technology  tools  provide  for  accomplishing  simultaneous  work  for  different  purposes.  The  amount  of 
time  that  students  take  to  undergo  understanding,  research  and  resolution  differs  and  therefore  designers  need  to 
account  for  this.  It  also  has  implications  for  facilitator  increased  or  decreased  intervention  per  stage  as  need. 

Cognitive  Tools 

Technology  tools  that  support  PBL  experiences  can  also  be  explored  in  a dimension  that  goes  beyond 
their  technological  characteristics.  Orrill  (2002)  deals  with  a technology-based  tool  to  support  metacognition 
during  the  collaborative  inquiry  during  PBL.  This  tool  used  threaded  discussion  with  message  labeling  to 
promote  metacognition.  The  aim  of  the  development  of  the  tool  was  to  support  PBL  thinking  as  opposed  to 
mere  logistics  management.  The  focus  of  the  course  was  technology  integration  in  K-12  education.  The  length 
of  the  PBL  portion  of  the  course  was  three  weeks.  The  analysis  of  interactions  in  the  Asynchronous  Conference 
Tool  (ACT)  comprised  discussions  that  occurred  during  the  first  phase  of  the  project  (approximately  1 1 days). 
The  researcher  conducted  an  analysis  of  the  character  of  interactions:  whether  messages  focused  overall  on 
defining  the  problem  / discussing  issues  (e.g.  present  student  thinking,  asked  content -related  questions),  tasks 
(e.g.  verify  due  dates,  corroborate  aspects  of  assignment,  focus  in  course-related  aspects  of  the  problem)  or 
other  (e.g.  supportive  messages). 

Student  use  of  the  tool  as  a process  manager  provoked  less  complains  about  it  in  comparison  with  use 
to  directly  support  problem  solving  inquiry.  The  author  suggests  that  this  indicated  a more  limited  support  of  the 
tool  for  the  later  purpose.  She  concluded  that  it  was  apparent  that  PBL  can  be  successful  and  worthwhile  in 
distributed  learning  environments.  Nevertheless,  there  was  an  evident  need  for  a more  robust  system  for 
supporting  communication,  organization  of  resources  and  issue  development.  Recommendations  for  continued 
research  spanned  finding  ways  to  support  simultaneously  online  PBL  and  its  process  management,  and  design 
considerations  on  how  to  structure  the  discussion  space  for  promoting  meaningful  conversation  which  is  critical 
to  success. 

Assessment 

Assessment  is  an  issue  that  demands  special  consideration.  Constructivist  approaches  to  assessment 
suggest  embedding  it  in  the  learning  experience  and  incorporating  students  as  designers  of  the  assessment 
mechanisms  as  well.  Sage’s  (2000)  findings  indicate  that  educators  need  to  consider  individual  and  group 
performance  in  the  online  environments.  Taplin  (2000)  also  brings  into  consideration  the  importance  of 
rethinking  assessment  in  the  online  problem-based  learning  environment.  She  suggests  further  research  to 
investigate  effective  ways  to  evidence  knowledge  construction  within  this  setting. 

Conclusions 

Several  authors  assert  that  distance  learning  environments  seem  more  prone  to  implement 
constructivist  principles  (Crumpacker,  2001;  Orrill,  2002;  Poon  et  ah,  1997).  Several  benefits  are  highlighted. 
Learners  from  around  the  world  could  work  together  benefiting  from  their  multiple  perspectives.  Continuous 
collaborative  work  could  be  guaranteed  even  if  some  group  members  are  temporarily  unavailable  for  physical 
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proximity  due  to  job-related  or  other  circumstances.  Its  resemblance  to  workplace  problem-solving  is 
highlighted  by  its  advocates.  Conversely,  Taplin  (2000)  considers  divergent  opinions  about  its  feasibility.  The 
lack  of  physical  proximity  and  challenges  in  student  support  are  reasons  proposed  by  those  who  do  not 
necessarily  support  this  approach  in  online  learning.  The  transition  from  face-to-face  to  online  PBL  has  been 
regarded  as  having  “obstacles”  (p.  41)  that  need  to  be  addressed  (Orrill,  2002). 

Because  of  the  innovation  of  the  combination  of  distance  learning  context  and  problem-based 
approach,  McConnell  (2002)  asserts  that  course  designers  and  tutors  will  need  to  understand  its  implications  for 
learning  and  teaching.  This  author  claims  that  it  is  a complex  and  yet  little  understood  form  of  distributed 
learning.  Therefore,  he  makes  the  case  for  ongoing  research  that  is  “exploratory,  descriptive,  grounded  in  real 
learning  situations  and  contexts,  addressing  both  broad  themes  and  micro  issues”  (p.  80)  to  help  increase 
understanding.  Sage  (2000)  briefly  explores  the  possibility  of  the  need  for  differentiated  forms  of  online  PBL 
tuned  to  the  nature  of  their  contexts.  Other  authors  explore  the  idea  of  the  need  for  understanding  online 
learning  as  it  constitutes  a different  enterprise  than  its  face-to-face  counterpart  in  its  very  nature  (Birnbaum, 
2001).  A current  of  educational  thinking  maintains  that  not  only  action  but  also  thought  are  reshaped  by  the 
tools  that  support  interaction  (Wertsch,  2002).  Communication  theory  also  supports  that  idea.  Computer- 
mediated  communication  within  online  PBL  implementations  may  be  such  an  agent  for  that  transformation. 
Therefore,  continuous  research  on  its  relevant  design  components  should  constantly  inform  practice. 
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Building  Bridge  between  Learning  and  Positive  Emotion:  How  to  Apply 
Emotional  Factor  in  Instructional  Designing  Process? 

Sanghoon  Park 
Florida  State  University 

Positive  emotions  and  Cognitive  process 

For  millennia,  emotional  states 1 have  been  viewed  as  avoidable  impediments  to  rational  thinking  (Ellis 
& Newton,  2000).  Several  reasons  have  been  pointed  out.  The  lack  of  consensus  of  the  definition  on  emotion 
that  tend  to  conflict  with  each  other  was  suggested  as  a main  reason  (Price,  1998).  Also  the  difficulty  of 
research  methodology  such  as  direct  observation  of  private  internal  emotional  experience  and  the  experiment 
setups  which  often  hide  away  the  true  nature  of  real  emotional  experience  were  indicated  as  additional  reasons 
(Gadanho  & Hallam,  2001).  Yet,  even  with  those  restrictions,  the  research  on  emotions  has  been  conducted  in 
various  academic  fields  including  politics,  business  administrations,  economics,  organizational  science, 
computer  science,  and  medical  science 

Except  the  case  of  medical  science,  the  emotion  related  researches  are  mainly  focused  on  the  effect  of 
emotion,  especially  positive  emotions  on  the  cognitive  process.  Positive  emotions  are  usually  considered  as 
“pleasant”,  but  the  others  are  considered  as  negative  (Gadanho  & Hallam,  2001).  This  series  of  researches  have 
been  providing  the  evidences  that  the  positive  emotions  has  a crucial  effect  on  diverse  cognitive  processes  such 
as  information  processing,  communication  processing,  negotiation  processing,  decision-making  processing, 
category  sorting  task  and  even  creative  problem  solving  process.  For  example,  Isen  and  Baron  (1991) 
summarized  based  on  the  program  of  empirical  researches  that  “persons  who  are  feeling  happy  are  more 
cognitively  flexible,  more  able  to  make  associations,  more  able  to  see  potential  relations  among  stimuli  than 
other  persons  in  a neutral  state”  (Isen,  Means,  1983;  Isen  & Daubman,  1984;  ; Isen,  Johnson,  Mertz,  & 

Robinson,  1985;  Isen,  Dubman,  & Nowicki,  1987).  In  her  Broaden-and-Build  Model  of  positive  emotions, 
Fredrickson  (1998)  identified  four  positive  emotions  such  as  joy,  interest,  contentment,  and  love,  and  further 
suggested  that  positive  emotions  broaden  the  scope  of  attention,  the  scope  of  cognition,  and  the  scope  of  action. 
The  theory  holds  that,  over  time,  the  broadening  triggered  by  positive  emotions  builds  a range  of  personal 
resources,  including  physical  resources  (e.g.,  physical  skills,  health,  longevity),  social  resources  (e.g., 
friendships,  social  support  networks),  intellectual  resources  (e.g.,  expert  knowledge,  intellectual  complexity), 
and  psychological  resources  (e.g.,  resilience,  optimism,  creativity)  (Fredrickson,  1998).  Forgas  (1998)  also 
found  that  person  in  a positive  mood  formulated  action  plans  that  were  more  cooperative  and  integrative,  and 
achieved  agreements  of  higher  quality  than  did  neutral  or  negative  mood  participants. 

On  the  other  hand,  there  are  some  other  researchers  who  argue  that  positive  emotion  and  negative 
emotion  plays  different  role  in  cognitive  process.  Bolte,  Goschke,  and  Kuhl  (2003)  insisted  the  positive  and 
negative  emotions  are  accompanied  by  qualitatively  different  information  processing  models  based  on  the 
personality  systems  interaction  theory  (Kuhl,  2000).  According  to  this  theory,  an  increase  in  positive  affect 
supports  a holistic  processing  mode,  which  is  characterized  in  memory  by  the  activation  of  wide  semantic 
fields,  which  include  weak  or  remote  association.  In  contrast,  an  increase  in  negative  affect  supports  an  analytic 
processing  mode,  which  is  characterized  by  a more  restricted  spread  if  activation  to  close  associates  and 
dominant  word  meanings.  Furthermore,  positive  mood  states  have  been  found  to  impair  some  aspects  of 
cognition,  causing  poor  performance  on  tasks  assessing  memory,  deductive  reasoning,  and  planning  (Oaksford, 
Morris,  Grainger,  & Williams,  1996;  Seibert  & Ellis,  1991) 

Emotion  in  instructional  design 

In  learning  context,  emotion  has  been  regarded  as  a relatively  less  significant  factor  affecting  on 
successful  learning  than  cognition  has,  even  though  there  have  been  some  critiques  insisting  that  cognitive 
theories  lack  an  adequate  conceptualization  of  the  impact  of  motivation  and  emotional  factors  in  learning 
(Csikszentmihalyi,  1990;  Krapp,  1992).  Emotion  is  not  generally  recognized  by  the  disciplines  that  address  the 
broad  issues  of  understanding  complex  systems  and  complex  behavior,  especially  in  the  presence  of  learning 
(Kort,  Reilly  & Picard,  2001).  However,  like  other  academic  field,  the  issue  of  emotion  has  investigated  without 
being  highlighted. 
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In  recent  times  especially,  the  research  on  emotion  in  learning  context  has  been  conducted  actively 
from  two  different  approaches.  One  approach  has  been  focused  on  fostering  affective  dimensions  of  human 
learning  and  development  by  designing  instruction  on  affective  domain  that  is  consisted  of  six  different 
categories  such  as  emotional  development,  moral  development,  social  development,  spiritual  development, 
aesthetic  development,  and  motivational  development  (Martin  & Reigeluth,  1999).  Emotion  is  proposed  as  one 
category  of  affective  domains  that  needs  to  be  developed  properly.  Emotional  development  includes 
understanding  own  and  other’s  feelings  and  affective  evaluations,  learning  to  manage  those  feelings,  and 
wanting  to  do  so  (Martin,  1999).  Bloom  (1964)  had  identified  three  domains  of  educational  activities  and 
described  affective  domain  as  five  major  categories  includes  the  manner  in  which  we  deal  with  things 
emotionally,  such  as  feelings,  values,  appreciation,  enthusiasms,  motivations,  and  attitudes.  This  series  of 
studies  is  called  as  “instructional  designing  in  affective  domain”.  The  studies  on  emotional  that  also  have  dealt 
with  key  questions  mentioned  above  are  in  the  same  vein  of  research  (Salvory,  1997). 

The  other  approach  of  emotion  related  researches  concentrated  on  how  to  moderate  emotions  that 
could  arise  during  the  learning  course.  These  kinds  of  study,  unlike  the  first  approach,  doesn’t  consider 
emotional  development,  but  try  to  integrate  learner’s  emotion  states  in  learning  context  aiming  at  how  to  handle 
learner’s  unstable  emotional  aspects  to  be  more  appropriately  maintained  during  entire  learning  course. 
Generally,  in  this  scope  of  researches,  emotions  are  assumed  being  scattered  on  some  position  from  positive 
emotions  to  negative  emotions  (Astleitner,  2000;  Astleitner,  2001;  Kort,  Reily  & Picard,  2001).  Astleitner 
(2000)  suggested  that  there  are  five  emotions  needed  to  be  considered  in  learning  context,  which  are  fear,  envy, 
anger,  sympathy,  and  pleasure.  He  added  that  instructional  designer  need  to  consider  those  emotions  to  optimize 
the  leaner’s  emotion  states  during  learning  process.  Kort,  Reilly  and  Picard  (2001)  also  proposed  model  relating 
phases  of  learning  to  emotions.  They  divided  emotions  into  positive  affect  and  negative  affect  according  to  the 
cognitive  dynamics  of  the  learning  process.  Therefore,  all  possible  emotions  will  be  allocated  in  four  different 
quadrant,  which  are  constructive  learning  - positive  affect  (e.g.,  awe,  satisfaction,  curiosity),  constructive 
learning  - negative  affect  (e.g.,  disappointment,  puzzlement,  confusion),  unlearning  - positive  affect  (e.g., 
hopefulness,  fresh  research),  finally  unlearning  - negative  affect  (e.g.,  frustration,  discard,  misconceptions). 

This  model  assumed  that  there  is  a process  of  specific  emotions  and  equivalent  cognition  in  learning  context 
and  those  emotions  are  derived  from  30  different  emotion  states. 

From  the  viewpoint  of  general  emotion  related  research,  the  problems  are  twofold.  First,  it  is  not  quite 
easy  to  figure  out  what  emotions  have  to  be  considered  in  learning  context.  Second,  it  is  also  questionable  how 
the  emotions  could  be  categorized  to  help  instructional  designer  utilize  emotions  into  actual  designing  of 
learning  process.  Regarding  the  first  question,  Ekman  (1999)  provided  several  characteristics  that  basic 
emotions  have  to  contain  in  general.  However,  figuring  out  the  basic  emotions  in  learning  context  is  not  a main 
concern  to  an  instructional  designer,  because  not  only  basic  emotions  but  also  other  related  emotions  can  be 
possibly  occurred  during  learning  context.  As  mentioned  previously,  how  to  optimize  learner’s  emotion  states  is 
the  key  question  for  instructional  designer. 

Considering  the  previous  researches  on  the  effects  of  positive  emotion  on  the  cognitive  process,  it 
might  be  one  reasonable  way  to  categorize  the  emotions  into  positive  emotions  and  negative  emotions,  since  it 
is  quite  obvious  to  differentiate  positive  emotions  from  negative  emotions.  As  a matter  of  fact,  this  classification 
of  emotion  has  been  broadly  used  in  the  research  area  of  motivation  and  emotion  in  learning  context. 

To  clarify  the  definition  of  positive  emotions  in  the  learning  context,  we  need  to  understand  two  facts. 
One  is  the  context  of  positive  emotion,  and  the  other  is  the  characteristics  of  positive  emotion.  First,  the  positive 
emotion  has  to  be  considered  in  the  context  of  learning.  Referring  the  emotion  sets  in  learning  context  shows 
that  there  are  several  types  of  positive  emotions  such  as  confidence,  intrigue,  epiphany,  enthusiastic,  excited, 
hopeful,  curiosity,  enlightenment,  thrilled,  anticipatory,  comfort,  interest,  insight,  satisfied  and  calm  (Kort, 

Reilly  & Picart,  2001).  Those  emotions  are  assumed  to  be  helpful  for  learner  to  concentrate  on  the  learning. 
However,  we  have  to  be  careful  with  saying  that  “All  of  these  positive  emotions  that  we  can  verbalize  have  to 
be  considered  in  instructional  designing  process!”,  because  again,  we  are  faced  with  the  critical  question,  what 
are  the  positive  emotions.  Thus,  the  characteristics  of  emotion  also  need  to  be  taken  into  account  in  defining 
positive  emotions.  In  designing  robot  learning  context,  Gadanho  and  Hallam  (2001)  described  some  of  the 
characteristics  of  emotion  in  learning  process.  First,  emotion  have  valence  that  is,  they  provide  a positive  or 
negative  value.  Second,  emotions  have  some  persistence  in  time  that  is,  it  is  not  allowed  to  have  sudden 
unrealistic  swings  between  different  emotions,  particularly  when  the  emotions  in  question  differ  a lot.  Finally, 
emotions  color  perception  in  that  what  is  perceived  is  biased  by  the  current  emotional  state. 

Given  those  two  facts,  positive  emotions  are  well  understood  as  the  emotions  which  help  learner 
concentrate  on  the  learning  task,  which  does  not  swing  to  different  emotion  suddenly,  and  which  can  be  affected 
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by  previous  emotional  states.  For  example,  when  the  learner  is  not  familiar  with  the  learning  task  or  contents, 
he/she  will  probably  feel  confused  or  frustrated  at  beginning  of  the  learning.  This  emotion  states  will  be 
automatically  changed  to  the  anxiety,  if  the  task  is  not  optional  but  required  step  for  the  learning.  However,  if 
the  learner  is  getting  knowledgeable  with  the  specific  tasks  or  contents,  he/she  will  feel  comfortable  and 
furthermore,  be  satisfied  with  what  he/she  is  doing,  finally  be  enjoy  working  on  the  learning  tasks.  Of  course, 
these  are  reasonably  explained  well  when  only  the  learning  context  is  considered.  There  might  be  a lot  of 
different  factors  which  affect  the  learner’s  emotional  states  such  as  instructor,  classroom  atmosphere,  weather, 
and  personal  issues.  However,  if  we  only  think  about  the  learning  context,  which  is  the  interaction  between 
learner  and  learning  task,  positive  emotions  will  be  one  of  obvious  predictors  that  can  positively  affect  learner’s 
motivation  and  even  performance. 

Two  positive  emotion  states  and  instructional  design 

There  has  been  several  emotion  states  suggested  in  the  research  of  motivation  field.  The  researches  on 
motivation  have  been  conducted  from  diverse  topics  such  as  expectancy,  control,  engagement,  interest,  goal, 
attribution,  value,  self-worth,  self-regulation,  and  volition  (Eccles  & Wigfield,  2002).  Among  those  themes, 
intrinsic  motivation-related  topics  are  mostly  covering  the  emotion  issues  . When  individuals  are  intrinsically 
motivated,  they  participate  in  activity  because  of  learning  interest  and  activity  enjoyment  (Eccles  & Wigfield, 
2002).  For  example,  Deci  and  Ryan  (1985)  proposed  self-determination  theory  in  which  two  perspectives  on 
human  motivation  are  integrated,  one  is  the  optimal  level  of  stimulation  and  the  other  is  basic  needs  for 
competence.  Csikszentmihalyi  (1990)  also  defined  the  intrinsic  motivation  with  the  concept  of  “Flow”,  which 
describes  experience  of  fully  engagement  in  terms  of  an  emotional  state.  Keller  (1983)  also  suggests  some 
emotional  aspect  with  his  ARCS  model.  According  to  his  ARCS  motivational  design  model,  attention  included 
perceptual  arousal  and  inquiry  arousal.  However,  the  other  components  of  ARCS  model  contain  also  some 
degree  of  cognitive  process.  Among  those  theories  and  models,  learning  interest  and  flow  has  been  thought  as 
an  important  part  of  intrinsic  motivation  (Eccles  & Wigfield,  2002).  Since  the  motivation  is  considered  as  the 
combined  construct  between  emotional  process  and  cognitive  process  together,  it  is  very  difficult  to 
differentiate  the  former  from  the  latter.  However  both  of  the  interest  theory  and  flow  theory  put  an  emphasis  on 
the  emotional  aspect  of  motivation.  Interest  also  has  been  considered  as  one  of  the  positive  emotions  in  learning 
and  life  (Kort,  Reilly  & Picard,  2001;  Fredrickson,  2001). 

Learning  interest 

Among  the  various  emotions  that  have  been  studied  in  learning  context,  the  role  of  interest  and  its 
implications  for  learning  was  studied  from  the  beginning  of  the  20th  centuries  (Dewey,  1913).  Since  then,  there 
have  been  a relatively  large  number  of  new  empirical  studies  concerned  with  both  the  influence  of  interest  on 
learning  and  development  and  the  origin  and  transformation  of  interests  (Krapp,  Hidi  & Renninger,  1992).  The 
studies  conceptualized  interest  in  a variety  of  ways  based  on  research  questions  and  methods  they  had  used  in 
psychological  or  educational  setting. 

However,  most  common  assumption  of  the  concepts  is  that  interest  is  a phenomenon  that  emerges  from 
an  individual’s  interaction  with  his  or  her  environment  (Krapp,  1992).  Based  on  the  common  assumption, 
psychologist  considered  the  characteristics  of  interest  as  followings  (Izard  & Ackerman  2000).  First,  Interest 
motives  exploration  and  learning,  and  guarantees  the  person's  engagement  in  the  environment.  Second,  Interest 
is  the  only  emotion  that  can  sustain  long-term  constructivist  or  creative  endeavors.  These  two  significant 
features  of  interest  imply  that  interest  and  motivation  has  very  strong  relationship.  For  example,  Keller’s  ARCS 
model  (1987)  contains  the  interest  as  one  of  the  motivational  aspects,  attention. 

In  learning  context,  the  issue  of  interest  has  been  investigated  under  different  name  such  as  epistemic 
curiosity,  perceptual  curiosity,  cognitive  interest  and  emotional  interest,  Berlyne  (1965)  suggested  distinction 
between  epistemic  curiosity  and  perceptual  curiosity.  According  to  his  distinction,  epistemic  curiosity  concerns 
enquiry  about  knowledge  and  is  shown  when  one  puzzles  over  some  science  problem.  Perceptual  curiosity 
concerns  increased  attention  given  to  objects  in  the  environment  such  as  symmetrical  figure.  He  concluded  that 
curiosity  related  to  work  in  school  setting  seems  to  belong  more  to  epistemic  rather  than  perceptual  curiosity. 

The  concept  of  cognitive  interest  and  emotional  interest  was  proposed  by  Kintsch  (1980).  According  to 
the  cognitive  interest,  cognitive  interest  adjuncts  such  as  explanative  summaries,  influence  learner’s  cognition 
by  promoting  the  reader  are  structural  understanding  of  the  explanation.  On  the  other  hand,  emotional  interest  is 
explained  by  that  the  addition  of  interesting  but  irrelevant  material  to  a textbook  lesson  energizes  learner  so  that 
they  pay  more  attention  and  learn  more  overall.  However,  the  results  of  the  two  experiments  about  emotional 
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interest  and  cognitive  interest  show  that  emotional  interest  doesn’t  influence  learner’s  actual  learning 
performance.  It  is  quite  attractive  because  those  two  researchers  tried  to  explain  two  similar  concepts  with  two 
different  names. 

Lot  of  researches  on  interest  in  reading  education  field  have  showed  that  two  viewpoints  of  interest  are 
assumed,  one  is  individual  interest  and  other  is  situational  interest  (Krapp,  Hidi  & Renninger,  1992). 

From  the  view  of  Individual  interest,  Interest  is  implied  as  a characteristic  of  person.  It  is  specific  to 
individuals,  developed  slowly,  tends  to  be  long  lasting,  and  triggered  by  individual’s  predisposition.  For 
example,  learners  who  are  interested  in  topic  or  an  activity  pay  more  attention  and  acquire  more  knowledge  that 
participants  without  such  interest.  Therefore,  it  is  extremely  time  intensive  and  effort  consuming  to  design 
learning  environment  in  terms  of  individual  interest.  Finding  out  every  possible  individual  interest  and  defining 
it  to  design  learning  environment  are  huge  challenge. 

Situational  interest  is  generated  as  a result  of  interestingness.  It  is  caused  primarily  by  certain 
conditions  and  concrete  objects  in  the  environment,  triggered  by  environmental  factors,  possibility  of 
prescription,  elicited  by  certain  aspects  of  a situation,  and  it  is  assumed  to  contribute  to  the  interestingness  of  the 
situation. 

In  summary,  individual  interest  is  a relatively  stable  evaluative  orientation  towards  certain  domains,  and 
situational  interest  if  an  emotional  state  aroused  by  specific  features  of  an  activity  or  a task. 

The  (Figure  1)  illustrates  the  relationship  between  individual  interest  and  situational  interest.  As 
described,  individual  interest  also  can  be  affected  by  two  aspects,  feeling-related  valence  and  value-related 
valence  (Schiefele,  1999).  Feeling-related  valences  refer  to  the  feelings  that  are  associated  with  an  object  or  an 
activity,  for  instance,  involvement,  stimulation,  and  flow.  Value  related  valences  refer  to  the  attribution  of 
personal  significance  of  importance  to  an  object  to  and  activity.  Since  those  two  aspects  are  highly  correlated 
each  other,  it  is  hard  to  say  from  which  aspect  the  learner  feel  interested  in.  However,  it  is  obvious  that  some 
learners  do  the  activity  because  they  like  it  primarily  based  on  feeling,  and  others  do  the  activity  because  they 
think  the  activity  is  important  for  other  purpose.  Most  of  the  researches  on  situational  interest  have  focused  on 
the  characteristics  of  academic  tasks  that  create  interest  (Hidi  & Baird,  1986).  Following  features  have  been 
found  to  arouse  situational  interest  and  promote  text  comprehension  and  recall:  personal  relevance,  novelty, 
activity  level,  and  comprehensibility. 


Figure  1.  Individual  interest  and  Situational  interest 
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According  to  the  distinction  between  cognitive  interest  and  emotional  interest,  those  are  interest  state 
derived  by  specific  feature  of  a text.  Therefore,  both  of  cognitive  interest  and  emotional  interest  will  be  counted 
as  aspects  of  situational  interest. 


Flow 

According  to  Watson  et  al  (1988),  positive  affect  refers  to  the  extend  to  which  an  individual  feels 
active,  enthused  and  alert.  It  also  reflects  emotional  states  such  as  enthusiasm,  drive,  alertness,  interest, 
joyousness,  being  self-determined,  and  therefore  the  ability  of  the  individual  to  enjoy  his  or  her  surroundings, 
make  use  of  and  enjoy  given  opportunities  (Konradt,  Filip,  & Hoffman,  2003).  As  already  mentioned  in  the  first 
paragraph  of  this  paper,  positive  affect  support  cognitive  flexibility,  which  means  that  person  in  a positive  affect 
are  superior  in  tasks  requiring  creativity  in  comparison  to  subjects  being  negatively  aroused. 

Flow  is  a state  of  experience  which  is  characterized  by  an  experience  of  intense  concentration  and 
enjoyment  (Csikszentmihalyi,  1990).  When  people  reflect  on  how  it  felt  when  they  were  in  flow,  they  mention 
at  least  one,  and  often  all,  of  these  aspects:  (a)  sensing  that  one’s  skills  are  balanced  by  challenge,  (b)  engaging 
in  a goal-directed  activity,  (b)  receiving  clear  feedback,  (d)  feeling  in  control,  (e)  intensifying  concentration, 
within  a sense  of  (f)  merging  awareness,  (h)  disappearing  of  self-consciousness,  (i)  distorting  the  sense  of  time, 
and  (j)  perceiving  the  experience  as  intrinsically  rewarding  (Csikszentmihalyi,  1990). 

In  learning  context,  the  term  “flow”  is  used  to  refer  to  “optimal  experience”  events.  The  earliest 
writings  on  flow  have  signaled  the  expectation  that  flow  is  particularly  important  concept  is  an  educational 
setting.  It  is  also  supported  by  the  more  recent  researches  showing  that  flow  occurred  more  often  during  study 
and  schoolwork  than  other  daily  activities  (Massimini  & Carli,  1988). 

The  flow  state  can  be  represented  as  a “channel”  on  a plot  of  challenge  versus  skills,  separating  the 
states  of  anxiety  and  boredom  (Figure  2). 


Figure  2.  Csikszentmihalyi’ s original  model  of  flow 

According  to  this  model,  only  a relative  balance  of  challenge  and  skill  is  relevant  to  flow,  not  the 
absolute  values.  That  is,  if  the  challenge  of  a task  decreases,  it  might  become  boring.  However,  if  the  challenge 
increases  but  one’s  skills  do  not  improve  to  meet  the  challenge,  then  one  might  get  into  a state  of  anxiety.  A 
learning  activity  might  produce  a progression  up  to  the  flow  channel  as  new  skills  are  learnt  and  greater 
challenges  are  sought  on  which  to  exercise  those  skills  (Csikszentmihalyi,  1990).  This  model  has  been  refined 
to  include  four,  eight  different  states  with  the  concept  of  channel. 
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Figure  3.  Four  channel  model  of  flow  (Massimini  & Carli,  1988) 


Figure  4.  Eight  channel  model  of  flow  (Massimini  & Carli,  1988) 

Recently,  a series  of  researches  have  been  conducted  to  apply  the  concept  of  flow  in  instructional 
technology  field  (Chen,  Wigand,  & Nilan,  1998;  Chen,  Wigand,  & Nilan,  1999;  Chan,  Repman,  1999;  Konradt 
& sulz,  2001;  Konradt,  Filip,  & Hoffman,  2003).  Since  the  concept  of  flow  has  been  defied  as  a optimal 
experience,  which  is  the  balance  of  challenge  and  skill  of  learner,  these  studies  tried  to  investigate  how  the 
learner’s  experience  have  changed  while  they  are  working  on  the  tasks  in  web  environment  and  hypermedia 
environment. 

For  measuring  methods  of  flow,  several  approaches  has  been  suggested  and  employed.  First,  ESM 
(Experience  Sampling  Method)  was  developed  in  which  respondents  were  electronically  paged  about  8 times  a 
day  for  a week  to  prompt  them  to  a questionnaire  (Csikszentmihalyi  & Larson,  1987).  A digital  implementation 
of  the  ESM  also  developed  and  used  in  every  day  web  activity  (Chen  & Nilan,  1998).  To  measure  overall  state 
of  flow,  FSS  (Flow  Status  Survey)  also  has  been  developed  and  validated  (Chan  & Repman,  1999). 

An  ongoing  issue  in  measuring  flow  is  to  find  a method  which  makes  it  possible  to  measure  flow 
independently  from  the  positive  states  of  consciousness  such  as  happiness,  concentration,  control,  lack  of  self- 
consciousness,  and  lack  of  distraction.  One  way  was  to  use  a measure  of  the  balance  between  the  challenge  of 
an  activity  and  the  learner’s  perception  of  their  own  skill  to  do  the  activity.  These  two  variables  have  been 
reported  to  be  reliable  indicators  for  measuring  flow  (Novak  & Hoffman,  1997). 


^In  this  paper,  the  term  emotion  has  been  used  to  specify  the  feeling  states  which  refer  to  specific  states  for  the  specific 
object.  Mood  usually  refers  to  more  general  feeling  states  without  any  specific  object.  Affect  refers  to  more  stable, 
dispositional  feeling  states.  However,  researchers  usually  use  the  term  mood  and  affect  without  clear  distinction. 

2 

The  plural  form  of  emotion  (emotions)  was  used  instead  of  the  singular  form  of  emotion  (emotion),  because  there  can  be 
more  than  one  positive  emotion  and  negative  emotion  depending  on  the  context.  For  example,  Fredrickson  (1998)  chose  joy, 
interest,  contentment,  and  love  as  positive  emotions.  Ekman  (1992)  used  the  term  “emotional  families”  to  explain  this. 
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Abstract 

The  purpose  of  this  paper  was  to  investigate  the  effects  of  different  types  of  visual  illustrations  on 
learner’s  learning  interest , motivation  and  achievement,  especially  in  multimedia  learning.  The  participants 
were  drawn  from  two  classes  of  an  “Introduction  to  Educational  Technology”  course  and  randomly  assigned 
to  one  of  the  three  treatments:  one  with  cognitive  interest  illustrations  which  were  designed  to  signal  the 
structure  of  the  explanation  and  another  with  emotional  interest  illustrations  which  are  interesting  but 
irrelevant  illustration  to  understand  the  structure  of  text.  The  last  one  only  contained  text  information.  Result 
revealed  that  the  post  interest  was  different  between  learners  in  cognitive  interest  illustration  group  and  text- 
only  group,  and  also  different  between  learners  in  emotional  interest  illustration  group  and  text-only  group. 
However,  the  types  of  illustration  didn 't  have  an  effect  on  learner’s  achievement  in  terms  of  information  recall 
and  achievement  test.  There  were  also  no  significant  differences  between  learner’s  motivations  among  three 
illustration  groups.  The  features  of  the  type  of  illustration  that  may  have  contributed  to  the  findings  are 
discussed. 


Introduction 

The  effective  use  of  graphical  illustration  in  designing  instructional  material  has  been  suggested  as  an 
important  facet  of  instructional  message  design  (Anglin,  Towers,  and  Levis,  1996).  Using  illustration  in 
instructional  material  is  an  effective  method  to  support  learning  because  it  can  be  used  as  interest-getting  device 
and  it  also  helps  learner  interpret  and  remember  the  context  of  illustrated  text.  Since  Spaulding  (1955)  reviewed 
sixteen  research  studies  on  the  topic  of  using  illustration  in  instruction  conducted  between  1930  and  1953,  many 
researchers  have  studied  the  effects  of  illustration  on  knowledge  acquis  ition  in  instructional  setting  (Samuels, 
1970;  Holiday,  1973;  Concannon,  1975;  Schallert,  1980;  Levi  & Lentz;  1982,  Brody,  1984;  Mayer,  1989;  Levin 
& Mayer,  1993;  Mayer,  Steinhoff,  Bower,  & Mars,  1995;  Harp  & Mayer,  1997).  After  reviewing  those  studies, 
Anglin,  Towers  and  Levis  (1996)  summarized  that  visual  illustrations  can  facilitate  the  acquisition  of 
knowledge  when  they  are  presented  with  text  material  concurrently. 

However,  it  has  been  pointed  out  that  the  results  of  illustration  related  researches  can  not  be  integrated 
across  all  studies  because  of  the  lack  of  connections  in  terms  of  function  of  the  illustration  in  the  instructional 
treatment  (Anglin,  Towers,  and  Levis,  1996).  It  means  that  each  illustration  related  study  was  focusing  on  the 
one  function  of  illustration  that  might  be  different  from  the  function  of  other  illustration  related  research.  In 
order  to  avoid  the  generalization  of  the  results  of  illustration  related  researches,  the  functional  framework  has 
been  suggested.  The  functional  frame  work  provides  assistance  in  classifying  visual  illustrations  into  meaningful 
functional  categories.  Using  this  framework,  the  research  results  can  be  combined  and  generalized  differently 
depending  on  the  function  of  illustration.  Therefore  it  is  critical  to  determine  the  specific  function  of  illustration 
before  conducting  actual  research  on  the  effect  of  visual  illustration. 

Regarding  the  function  of  illustration,  Levie  and  Lantz  (1982)  suggested  a functional  framework  that 
includes  classifying  illustrations  in  text  based  on  how  they  impact  a learner.  According  to  them,  framework 
contains  four  major  functions  which  are  attentional,  affective,  cognitive,  and  compensatory  (Levie  & Lentz, 
1982).  The  attentional  function  attracts  or  directs  attention  to  the  material.  The  affective  function  enhances 
enjoyment  or  affects  emotion  and  attitude.  The  cognitive  function  serves  to  facilitate  learning  text  content 
through  improving  comprehension,  increasing  retention,  or  providing  additional  information.  The  last  function, 
the  compensatory  function,  is  used  to  accommodate  poor  readers.  Among  those  four  functions,  the  previous 
researches  usually  were  focusing  on  only  the  cognitive  function  of  illustration.  It  has  not  been  answered  clearly 
what  effect  the  visual  illustration  have  on  learner’s  emotion  and  attitude  and  how  the  illustration  need  to  be 
designed  to  improve  affective  function  of  illustration. 

In  terms  of  affective  function  of  illustration,  Kintch  (1980)  insisted  that  the  visual  illustration  has  an 
effect  on  the  learner’s  affective  status  in  two  different  ways  depending  on  the  features  of  illustration.  According 
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to  his  cognitive  interest  and  emotional  interest  theory,  visual  illustration  improves  learner’s  cognitive  interest 
and  it  promotes  learner’s  emotional  interest  as  well  depending  on  the  characteristic  of  it.  The  cognitive  interest 
influences  learner’s  cognition  by  promoting  the  reader’s  structural  understanding  of  the  explanation.  On  the 
other  hand,  emotional  interest  is  explained  by  the  addition  of  interesting  but  irrelevant  material  to  a textbook 
lesson.  It  energizes  learner’s  arousal  so  that  they  pay  more  attention  and  learn  more  overall.  Therefore, 
cognitive  interest  illustration  is  defined  as  the  illustration  that  signals  the  structure  of  the  explanation.  The 
emotional  interest  illustration  is  defined  as  the  illustration  which  refers  to  interesting  but  irrelevant  illustration 
to  understand  the  structure  of  text.  However,  it  plays  an  important  role  from  the  motivational  aspect,  because  it 
increases  emo  tional  arousal  and  further  influences  the  leader’s  cognitive  process. 

In  a same  vein,  Levie  and  Lentz  (1982)  reviewed  previous  researches  comparing  three  separate 
research  area  concerning  the  role  of  illustration  in  learning.  The  first  research  area  was  concerning  learning 
illustrated  text  information,  the  second  was  about  learning  nonillustrated  text  information,  and  the  last  was 
about  learning  using  a combination  of  illustrated  and  nonillustrated  text  information.  According  to  the  cognitive 
interest  theory,  the  illustrated  text  information  is  similar  with  the  cognitive  interest  illustration.  Also,  the 
nonillustrated  text  information  is  comparable  to  the  emotional  interest  illustration.  They  concluded  that  learning 
would  be  facilitated  when  the  information  in  the  written  text  is  depicted  in  the  illustrations  and  learning  of  text 
material  would  not  be  helped  nor  necessarily  hindered  with  illustrations  that  are  not  related  to  the  text. 

However,  they  didn’t  consider  the  affective  function  of  illustration  separately. 

Harp  and  Mayer  (1997)  examined  the  effects  of  emotional  interest  adjunct  and  cognitive  interest 
adjunct  on  information  retention,  learning  transfer,  and  learning  interest.  They  reported  that  learners  in  the  base 
text  group  recalled  the  most,  whereas  learners  who  read  passages  containing  the  base  text  along  with  emotional 
interest  text  and  illustrations  recalled  the  least.  This  result  was  consistent  with  the  prediction  of  cognitive 
interest  theory  and  inconsistent  with  the  prediction  of  emotional  interest  theory.  However,  the  learners’ 
achievement  was  measured  based  on  the  procedural  information  of  scientific  phenomenon,  not  based  on  the 
factual  information.  In  addition,  the  instructional  material  was  paper  based  material  and  the  participants  were 
not  allowed  to  read  the  passage  more  than  once. 

Therefore,  two  important  issues  can  be  raised.  The  first  issue  is  that  it  is  doubtful  if  the  instructional 
material  is  delivered  through  multimedia  presentation.  In  the  multimedia  learning,  courseware  screens  are 
consisted  of  text,  graphic  such  as  still  illustration  and/or  animation,  and  video  clip.  The  learners  can  navigate 
each  screen  through  the  navigation  buttons.  Thus  learner  can  go  back  to  previous  contents  text  or  go  to  next 
contents  text.  They  are  also  allowed  to  read  the  text  more  than  once.  In  other  word,  the  learner  has  control  over 
the  process  of  learning  in  multimedia  material  setting  rather  than  paper  based  material  setting.  The  second  issue 
is  that  the  achievement  result  can  be  different  depending  on  instructional  material  contents.  If  it  contains  factual 
information  as  well  as  procedural  information,  the  learners  in  emotional  interest  illustration  group  may  recall 
more  concepts  than  that  of  the  cognitive  interest  illustration  group,  because  the  learner  in  the  cognitive  interest 
illustration  group  will  not  be  able  to  get  any  benefits  from  studying  the  instructional  material  containing  the 
illustration  representing  the  procedural  information. 

In  order  to  answer  those  two  questions,  this  study  was  designed  with  learner  controlled  multimedia 
material  containing  factual  information  as  well  as  procedural  information.  Therefore  the  purpose  of  this  paper 
was  to  investigate  the  effects  of  different  types  of  visual  illustrations  on  learner’s  learning  interest,  motivation 
and  achievement. 

One  independent  variable,  “Visual  illustration  type’’  containing  three  types,  was  implemented  for  the 
research.  Two  different  types  of  illustration  were  applied  into  designing  instructional  material  from  the 
perspective  of  cognitive  interest  and  emotional  interest.  The  first  type  was  cognitive  interest  illustration  and 
text.  The  second  type  was  emotional  interest  illustration  and  text.  The  third  type  was  text  -only  material  without 
any  illustrations,  which  was  the  control  group.  The  experimental  group  and  control  group  were  formed  as 
follows.  (1)  Group  1 contained  text  information  and  cognitive  interest  illustration,  (2)  Group  2 contained  text 
information  and  emotional  interest  illustration,  (3)  Group  3 contained  only  text  information. 

Four  dependent  variables  were  examined.  The  first  dependent  variable  was  “Learning  interest.”  The 
learning  interest  was  measured  by  one  question,  which  adopted  from  the  research  by  Harp  and  Mayor  (1997). 
The  second  variable,  motivation  was  measured  by  IMMS  (Instructional  Material  Motivational  Survey) 
developed  based  on  Keller’s  ARCS  model.  IMMS  contains  question  items  to  measure  the  motivation  status 
from  four  different  aspects  such  as  attention,  relevance,  confidence,  and  satisfaction.  The  third  dependent 
variable  was  “Learner’s  concept  recall”  targeted  to  measure  the  number  of  factual  information  recalled  in 
limited  time.  The  last  variable  was  “Learner’s  achievement”  developed  to  measure  the  procedural  information 
using  achievement  test  on  the  topic  of  the  instructional  material  contents. 
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Since  the  learner’s  prior  topic  interest  and  prior  knowledge  were  assumed  as  important  covariates  that  could 
affect  on  learning  interest  and  learner’s  achievement,  those  prior  topic  interest  and  prior  knowledge  were  also 
measured  before  conducting  research  and  considered  as  two  covariates  in  data  analysis  stage. 

Given  the  characteristics  derived  from  the  definition  of  cognitive  interest  and  emotional  interest,  the 
learners  in  the  emotional  interest  illustration  group  were  predicted  to  show  higher  learning  interest  and 
motivation  than  those  of  the  learners  in  the  cognitive  interest  group  and  text -only  group  because  of  the  affective 
function  of  illustration.  The  first  hypothesis  is  justified  because,  according  to  the  definition,  emotional  interest  is 
derived  from  the  interesting  illustration  that  boosts  learner’ s arousal  level.  Therefore  the  learner  pays  more 
attention  on  the  instructional  material  and  is  motivated  overall  as  well.  In  addition,  the  learner  in  the  cognitive 
interest  illustration  group  no  longer  get  benefits  from  using  cognitive  interest  illustration,  because  the  learners  in 
all  three  groups  have  control  over  the  learning  material.  In  other  words,  learners  can  study  the  instructional 
material  mo  re  than  once  as  the  instructional  material  allows  learners  to  navigate  the  screens  using  buttons. 

The  learners  in  the  emotional  interest  illustration  group  were  also  expected  to  show  higher  score  in 
both  of  the  recall  test  and  achievement  test,  because  of  the  same  reason.  The  learner  will  pay  more  attention  to 
the  instructional  material  than  the  two  other  groups  so  that  the  learners  would  be  able  to  recall  more  concepts 
and  achieve  higher  score  than  those  of  the  two  other  groups. 

Method 


Participants 

The  participants  in  the  study  were  36  college  level  students  who  were  attending  a four-year  university 
in  southeastern  area  in  United  States.  The  study  took  place  during  the  “Introduction  to  Educational  Technology” 
course  which  designed  to  teach  how  to  apply  the  technology  into  learning  and  teaching.  HyperStudio,  one  of  the 
multimedia  authoring  tools,  was  used  in  this  experiment  because  it  was  the  required  software  learners  need  to 
study  in  the  class.  Four  of  the  learners  were  male  and  thirty  two  of  the  learners  were  female.  All  of  the  learners 
were  sophomore  and  junior  level  learners  who  were  willing  to  apply  to  the  college  of  education.  The 
participants  were  drawn  from  two  classes  of  “Introduction  to  Educational  Technology”  and  randomly  assigned 
to  one  of  the  three  treatments.  Total  number  of  participants  was  thirty  six  excluding  four  learners  who  decided 
not  to  attend  in  this  research. 

Materials 

The  multimedia  instructional  material  was  developed  using  HyperStudio  to  teach  the  “Life  cycle  of 
hurricane.”  The  material  consisted  of  10  screens  containing  6 different  concepts  covering  (1)  Unit  overview, 

(2)  Origin  of  Hurricane,  (3)  Life  Cycle  of  hurricane  development,  (4)  Eye  and  Eyewall,  (5)  Hurricane  rotation, 
and  (6)  Hurricane’s  demise.  Even  though  the  material  was  designed  using  multimedia  authoring  tool 
‘HyperStudio’,  only  graphic  and  text  information  were  applied  in  order  to  prevent  learner  from  learning  contents 
with  other  variables  such  as  sound  or  animation.  The  applied  illustrations  were  all  coloied  graphics  describing 
the  hurricane  development  process.  The  three  types  of  instructional  material  were  developed  separately 
according  to  the  differences  among  three  independent  variable  types. 

Independent  variables 

The  independent  variable  used  for  this  study  included  the  type  of  visual  illustration  used  to  deliver 
learners  the  concept  of  the  life  cycle  of  hurricane.  The  first  level  of  the  independent  variable  was  cognitive 
interest  illustration.  The  second  level  of  the  independent  variable  was  emotional  interest  illustration.  The  third 
level  of  the  independent  variable  was  text -only  information  without  illustration. 

The  instructional  material  containing  cognitive  interest  illustration  consisted  of  a screen-based 
presentation  on  the  topic  of  the  life  cycle  of  hurricane.  The  design  of  the  cognitive  interest  illustration  was 
centered  on  the  Kintch  (1980)’s  cognitive  interest  theory,  According  to  Kintch,  the  cognitive  interest  influences 
learner’s  cognition  by  promoting  the  reader’s  structural  understanding  of  the  text  explanation.  Therefore  the 
cognitive  interest  illustrations  need  to  be  designed  to  signal  the  structure  of  the  explanation.  For  example,  Harp 
and  Mayer  (1997)  used  an  illustrations  with  explanative  summary  designed  to  promote  cognitive  interest  on 
lightning  process.  In  present  research,  cognitive  interest  illustrations  were  designed  to  improve  the 
understanding  of  the  four  development  stage  of  Hurricane,  the  required  ingredients  for  a hurricane,  and  the 
location  of  eye  and  eyewall  as  shown  in  [Appendix  A].  The  illustrations  were  positioned  right  after  text 
information.  Therefore  the  participants  were  able  to  read  the  text  first  and  look  at  the  illustration  next.  Total 
number  of  the  cognitive  interest  illustration  was  six;  each  illustration  was  placed  on  the  screen  #3,  #4,  #5,  #6, 
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#7,  and  #8  with  corresponding  text. 

The  instructional  material  including  emotional  interest  illustration  was  also  consisted  of  the  screens  on 
the  topic  of  the  life  cycle  of  hurricane.  The  design  of  the  emotional  interest  illustration  was  based  on  emotional 
interest  theory  insisted  by  Kintch  (1980).  According  to  the  theory,  the  emotional  interest  energizes  learner’s 
arousal  so  that  they  pay  more  attention  on  the  instructional  material.  Therefore  the  emotional  interest 
illustrations  are  interesting  but  irrelevant  illustration  to  understand  the  structure  of  text.  Harp  and  Mayer  (1997) 
used  emotional  interest  illustrations  to  compare  the  effect  of  it  with  the  cognitive  interest  illustration.  In  present 
research,  emotional  interest  illustrations  were  designed  to  improve  the  learner’s  arousal  of  the  four  development 
stage  of  Hurricane,  the  required  ingredients  for  a hurricane,  and  the  location  of  eye  and  eyewall  as  shown  in 
[Appendix  B],  As  the  case  of  cognitive  interest,  emotional  interest  illustrations  were  positioned  right  after  text 
information.  The  position  and  the  number  of  emotional  interest  illustrations  were  same  as  those  of  cognitive 
interest  illustrations. 

The  instructional  material  containing  only  text  information  consisted  of  a screen-based  presentation  on  the  same 
topic  as  cognitive/emotional  interest  illustration  material.  However,  it  didn’t  include  any  illustrations.  Learners 
had  a full  control  over  the  navigation  using  buttons  throughout  the  instructional  material. 

Dependent  variables 

Four  dependent  variables  for  the  study  included  a learner’s  post-interest,  motivation,  the  number  of 
recalled  concepts,  and  achievement. 

One  post-interest  question  item  was  used  to  measure  learners’  post  interest  level.  This  post-interest 
item  was  used  in  previous  research  designed  to  see  how  much  the  learners  feel  interest  on  the  contents  (Harp  & 
Mayor,  1997).  Participants  were  asked  to  respond  their  interest  level  about  instructional  material  by  selecting 
one  of  five  choices  ranging  from  “Not  at  all  true’’  through  “Very  true’’. 

Learner’s  motivation  was  measured  using  IMMS  developed  by  Keller.  The  survey  included  36  items 
intended  to  be  a situational  measure  of  learners’  motivational  reactions  to  instructional  material.  The  response 
scale  ranges  from  1 to  5 with  12  Attention  related  questions,  9 Relevance  related  questions,  9 Confidence 
related  questions,  and  6 Satisfaction  related  questions.  The  reliability  based  on  Cronbach’s  alpha  for  each 
subscale  and  the  total  scale  was  Attention:  .89,  Relevance:  .81,  Confidence:  .90,  Satisfaction:  .92,  and  Total: 

.96. 

The  recall  test  was  designed  to  assess  the  learner’s  ability  to  remember  hurricane  related  terms.  The 
recall  sheet  had  the  following  instruction  typed  at  the  top  of  the  page:  "Please  write  down  everything  you  can 
remember  from  the  passage."  Participants  were  allowed  to  write  down  as  many  as  concepts  they  can  recall  from 
the  instructional  material  in  5 minutes  limitation. 

The  achievement  was  measured  using  post  achievement  test.  The  achievement  test  was  designed  to 
assess  the  learner’s  ability  to  solve  the  given  problems  using  what  they  have  learnt  from  the  instructional 
material.  Total  number  of  item  was  ten  including  five  short  answer  items  and  five  multiple  choice  items  . Items 
were  designed  to  ask  following  topics;  (a)  Four  development  process  of  hurricane,  (b)  Meteorological  factors 
necessary  for  forming  hurricane,  (c)  Identification  of  hurricane  from  real  weather  pictures,  and  (d)  The  structure 
of  hurricane. 

Procedures 

The  instructional  material  was  presented  in  a computer  laboratory  with  24  individual  personal 
computers.  Participants  were  drawn  from  two  sections  of  introduction  to  educational  technology  course  and 
randomly  assigned  to  one  of  three  treatments  groups:  the  cognitive  interest  illustration  group,  and  the  emotional 
interest  illustration  group,  and  the  text -only  group.  Participants  were  asked  to  fill  out  the  prior  knowledge  and 
pre  interest  survey  before  processing  instructional  material.  Then  they  were  informed  that  they  would  be 
studying  multimedia  instructional  material  on  the  life  cycle  of  hurricane  and  that,  after  they  finished  reading, 
they  would  be  asked  a series  of  questions  about  what  they  have  read.  They  were  instructed  to  read  the  material 
carefully  in  their  normal  reading  rates.  Instructor  was  present  at  all  times  to  ensure  that  they  were  studying  only 
the  multimedia  material.  Each  participant  was  given  the  material  corresponding  to  his/her  treatment  group  and 
told  to  start  studying.  They  were  not  alio  wed  to  take  notes  or  refer  to  other  resource.  As  each  participant 
finished  studying  after  lOminutes,  the  experimenter  handed  the  post  interest  survey  inventory  to  fill  out  at  his  or 
her  own  rate.  After  completing  the  post  interest  inventory,  participants  were  given  the  recall  test.  The 
experimenter  collected  the  recall  sheet  after  six  minutes  had  passed.  Next,  participants  were  given  achievement 
test  sheets  and  allowed  10  minute  to  work  on  test.  After  final  sheet  had  been  collected,  the  participants  were 
given  IMMS  to  measure  motivational  level.  After  this,  the  participants  were  thanked  for  their  participation. 
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The  scoring  procedure  for  the  recall  test  and  achievement  test  were  administrated  as  follows . Recall 
test  was  scored  based  on  prepared  criteria.  Since  only  differences  among  three  treatments  was  the  type  of 
illustration  on  6 slides  out  of  10  slides,  the  information  recalled  from  only  those  6 slides  were  considered  in 
scoring  procedure.  When  learner  described  all  of  development  stages  (1st  stage  through  final  4th  stage)  of 
hurricane  on  recall  sheet,  each  stage  was  considered  as  1 point.  The  score  for  necessary  five  ingredients  for 
hurricane  also  scored  1 point  each.  Lastly,  description  regarding  eye  and  eyewall  was  computed  1 point  each. 
Therefore  total  score  ranged  from  0 through  1 1 . 

Achievement  test  was  graded  based  on  pre  determined  answer  sheet.  Total  number  of  questions  was 
10,  but  each  question  had  different  weight  depending  on  the  difficulty  of  question.  Since  there  are  correct 
answers  for  all  of  questions,  answer  sheet  was  prepared  based  on  the  information  from  hurricane  learning 
material.  Total  score  ranged  from  0 through  19. 


Results 


Table  1.  Means  (Adjusted  means)  and  standard  deviations  of  dependent  variables  across 
groups. 


Treatment 

groups 

DVs 

Maximum 

Score 

Cognitive  interest 
Illustration  (n=12) 

Emotional  interest 
illustration  (n=12) 

Text  -only 
(n=12) 

M (Adj . M) 

SD 

M (Adj.  M) 

SD 

M (Adj.  M) 

SD 

Post  interest* 

5 

3.75  (3.87) 

.96 

3.83  (3.87) 

.83 

3.33  (3.05) 

1.15 

Motivation 

Attention 

60 

40.83 

8.56 

44.58 

7.25 

37.58 

10.59 

Relevance 

45 

27.92 

5.43 

30.42 

5.68 

25.25 

6.98 

Confidence 

45 

37.00 

4.84 

37.17 

5.22 

34.75 

6.74 

Satisfaction 

30 

19.08 

5.20 

19.25 

4.58 

15.92 

5.43 

Total 

180 

124.83 

21.44 

131.42 

19.29 

113.50 

25.07 

Recall  score 

11 

5.67  (5.64) 

1.15 

4.75  (4.71) 

1.05 

4.83  (4.90) 

1.59 

Achievement 

19 

12.67  (12.78) 

2.77 

11.83  (12.00) 

3.16 

13.17  (12.88) 

1.90 

score 

* p < .05 

Table  2. 

The  analysis  of  covariance  summary 

Source 

Sum  of  Squares 

df 

Mean  Square  F Value 

Preinterest 

18.565 

1 

18.565 

42.381* 

GROUP 

4.340 

2 

2.170 

4.954* 

Error 

14.018 

32 

.438 

*pc.05 

Post  interest 

Post  interest  score  was  collected  using  one  interest  survey  question  item.  An  analysis  of  covariance 
(ANCOVA)  was  conducted  to  determine  whether  the  post  interest  scores  for  the  three  groups  differed  after 
adjustments  were  made  for  pre  interest  differences.  Pre  interest  was  served  as  a covariate.  Table  1 presents  the 
mean  as  well  as  the  adjusted  mean,  and  standard  deviation  for  post  interest  scores  across  all  three  treatment 
conditions.  With  alpha  set  at  .05,  and  a sample  size  of  36  ( 12  per  cell),  it  was  determined  that  the  power  for 
determining  moderate  effects  was  .54.  A review  of  scatterplot  for  post  interest  scores  revealed  no  serious 
violation  of  the  normality  assumptions  required  for  linear  regression  analysis.  Testing  for  the  assumption  of 
homogeneity  of  variance,  a Levene’s  test  revealed  appropriate  homogeneity  of  the  post  interest  scores,  F(2,33) 

= 1.63,  p=.211.  The  assumption  of  equal  regression  slopes  was  tested  and  found  tenable,  F(2,30)=2.09,  p>.05. 

The  ANCOVA  indicated  at  least  one  pair  of  means  was  significantly  different,  F(2,32)=4.95,  pc. 05.  A 
post  hoc  analysis  using  a bonferroni  multiple  comparison,  with  alpha  at  .05,  showed  that  the  post  interest  score 
for  the  cognitive  interest  illustration  group  (M=3.87)  was  significantly  higher  than  the  post  interest  score  for  the 
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text -only  group  (M=3.05).  Also  the  the  post  interest  score  for  the  emotional  interest  illustration  group  (M=3.87) 
was  significantly  higher  than  the  post  interest  score  for  the  text -only  group  (M=3.05).  However,  there  was  no 
score  difference  between  cognitive  interest  illustration  group  (M=3.87)  and  emotional  interest  illustration  group 
(M=3.87).  Table  2 presents  the  analysis  of  covariance  summary. 

Motivation 

Motivation  data  was  collected  using  IMMS  developed  by  Keller  (1993).  An  analysis  of  variance  was 
conducted  on  the  participant’s  ratings  of  motivation  level  in  terms  of  attention,  relevance,  confidence,  and 
satisfaction.  Table  1 presents  the  mean  and  standard  deviation  for  motivation  scores  across  all  three  treatment 
conditions.  With  alpha  set  at  .05,  and  a sample  size  of  36  ( 12  per  cell),  it  was  determined  that  the  power  for 
determining  moderate  effects  was  .54.  A review  of  scatterplot  for  post  interest  scores  revealed  no  serious 
violation  of  the  normality  assumption.  Testing  for  the  assumption  of  homogeneity  of  variance,  a Levene’s  test 
revealed  appropriate  homogeneity  of  the  post  interest  scores,  Attention,  F(2,33)  = .439,  p=.648;  Relevance, 
F(2.33)=.  181,  p=.836;  Confidence,  F(2.33)=2.217,  p=,125;  Satisfaction,  F(2,23)=.  151,  p=. 861;  Total, 
F(2,33)=.287,  p=,752). 

The  results  of  the  ANOVA  revealed  that  none  of  the  mean  score  difference  was  significant.  The  each 
sub  scale  of  motivation  score  did  not  differ  across  the  type  of  illustration  group.  Sub  scale  means  (with  standard 
deviations  in  parentheses)  for  group  1 through  group3  were,  40.83  (8.56),  44.58  (7.25),  37.58  (10.59)  for 
attention,  27.92(5.43),  30.42(5.68),  25.25(6.98)  for  relevance,  37.00(4.84),  37.17(5.22),  34.75(6.74)  for 
confidence,  and  19.08(5.20),  19.25(4.58),  15.92(5.43)  for  satisfaction,  respectively. 

Recall  test 

Recall  test  data  was  collected  by  grading  the  number  of  hurricane  related  terms  the  learner  wrote  down 
in  limited  time.  An  analysis  of  covariance  (ANCOVA)  was  conducted  to  determine  whether  the  recall  test 
scores  for  the  three  groups  differed  after  adjustments  were  made  for  prior  knowledge  differences.  Prior 
knowledge  was  served  as  a covariate.  Table  1 presents  the  mean  as  well  as  the  adjusted  mean,  and  standard 
deviation  for  recall  test  score  across  all  three  treatment  conditions.  With  alpha  set  at  .05,  and  a sample  size  of  36 
(12  per  cell),  it  was  determined  that  the  power  for  determining  moderate  effects  was  .54.  A review  of  scatterplot 
for  post  interest  scores  revealed  no  serious  violation  of  the  normality  assumptions  required  for  linear  regression 
analysis.  Testing  for  the  assumption  of  homogeneity  of  variance,  a Levene’s  test  revealed  appropriate 
homogeneity  of  the  recall  test  scores,  F(2,33)  = .899,  p=.417.  The  assumption  of  equal  regression  slopes  was 
tested  and  found  tenable,  F(2,30)=.622,  p>.05.  The  recall  test  score  did  not  differ  across  the  type  of  illustration 
groups,  F(2,32)=1.724,  p>.05.  According  to  the  second  hypothesis,  emotion  interest  group  was  expected  to 
show  higher  recall  score  than  those  of  cognitive  interest  group  and  text -only  group.  However,  the  result 
revealed  that  there  were  no  significant  differences  among  different  types  of  illustration  groups. 

Achievement  test 

Achievement  test  data  was  collected  by  grading  the  number  of  correct  answer  for  the  achievement  test. 
An  analysis  of  covariance  (ANCOVA)  was  conducted  to  determine  whether  the  achievement  test  scores  for  the 
three  groups  differed  after  adjustments  were  made  for  prior  knowledge  differences.  Prior  knowledge  was  served 
as  a covariate.  Table  1 presents  the  mean  as  well  as  the  adjusted  mean,  and  standard  deviation  for  recall  test 
score  across  all  three  treatment  conditions.  With  alpha  set  at  .05,  and  a sample  size  of  36  (12  per  cell),  it  was 
determined  that  the  power  for  determining  moderate  effects  was  .54.  A review  of  scatterplot  for  post  interest 
scores  revealed  no  serious  violation  of  the  normality  assumptions  required  for  linear  regression  analysis. 

Testing  for  the  assumption  of  homogeneity  of  variance,  a Levene’s  test  revealed  appropriate  homogeneity  of  the 
recall  test  scores,  F(2,33)  = 1.152,  p=.328.  The  assumption  of  equal  regression  slopes  was  tested  and  found 
tenable,  F(2,30)=.330,  p>.05.  The  achievement  test  score  did  not  differ  across  the  type  of  illustration  groups, 
F(2,32)=.394,  p>.05.  It  was  also  predicted  that  the  score  of  the  emotional  illustration  group  will  be  higher  than 
those  of  cognitive  illustration  group  and  text  only  group.  However,  the  result  showed  that  there  were  no 
significant  differences  among  different  types  of  illustration  groups. 

Discussion 

Supporting  the  primary  hypothesis  of  the  study,  the  mean  score  of  post  interest  of  learners  in  the 
emotional  interest  illustration  group  was  significantly  higher  than  the  text  -only  group.  In  addition,  the  mean 
score  of  post  interest  of  learners  in  the  cognitive  interest  illustration  group  was  significantly  higher  than  the 
text -only  group.  This  result  indicates  that  learners  who  were  given  illustrations  feel  much  interest  than  the 
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learners  who  were  given  only-text  information.  However,  there  was  no  significant  difference  between  the  mean 
score  of  cognitive  interest  illustration  group  and  the  mean  score  of  emotional  interest  illustration  group.  This 
finding  affirms  that  when  learner  is  given  either  cognitive  interest  illustration  or  emotional  interest  illustration, 
he/she  is  aroused  and  feels  positive  emotion,  interest,  in  the  instructional  material.  This  result  is  consistent  with 
the  findings  of  Harp  and  Mayer  (1997).  They  pointed  out  that  the  failure  to  find  differences  in  students'  ratings 
of  interestingness  raises  the  possibility  that  the  distinction  between  emotional  interest  and  cognitive  interest 
based  on  learner’s  evaluation  of  how  the  material  was  entertaining 

The  hypothesis  predicting  a motivational  effect  for  the  type  of  illustration  was  not  supported  by  data. 

As  shown  in  Table  1,  there  was  mean  differences  among  three  groups,  but  those  differences  were  not 
significant.  The  possible  explanation  for  the  type  of  illustration  failing  to  predict  motivation  involves  the  effect 
of  multimedia  program.  All  learners  receive  the  same  instruction  using  same  multimedia  program.  This  program 
was  the  topic  of  the  “Educational  technology”  class  on  experiment  day.  Therefore  it  is  possible  that  they  are  all 
motivated  to  the  instructional  material  regardless  of  the  type  of  illustration,  because  the  material  was  developed 
by  a new  program  which  learners  are  motivated. 

The  failure  of  type  of  illustration  to  affect  recall  and  achievement  test  can  also  be  explained  in  the 
same  grounds.  Unlike  the  findings  of  Harp  and  Mayer  (1997),  learners  are  allowed  to  navigate  the  instructional 
screens  with  free  in  this  multimedia  instructional  material.  Therefore  they  could  go  back  to  the  information 
screen  to  confirm  that  they  understand  the  information  correctly  and  illustration  did  not  play  a strong  role  to 
emphasize  the  information. 

There  were  several  limitations  to  the  findings  of  this  study.  First,  learner’s  individual  characteristic  regarding 
the  preference  to  the  illustration  was  not  considered.  Because  learner  did  not  have  a control  over  the  illustration 
type,  he/she  had  to  have  illustrations  along  with  the  text  information.  Second,  the  effect  of  multimedia  authoring 
program  was  not  considered.  Since  the  program  was  totally  new  to  participants  in  this  research,  all  participants 
could  be  motivated  on  the  same  level  no  matter  what  illustrations  they  were  given. 

Additional  research  is  needed  to  fill  the  gaps  in  our  understanding  of  the  interaction  between  learner’s 
characteristic  and  the  type  of  illustration  in  multimedia  setting.  Future  research  should  also  attempt  to  determine 
different  functions  of  illustration  in  terms  of  cognitive  function  as  well  as  affective  function.  Hence  it  will  be 
possible  to  compare  the  different  research  results  on  the  same  topic  based  on  the  function  of  illustration. 

The  implication  of  this  study  involves  the  importance  of  visual  illustration  on  learner’s  affect  in 
multimedia  learning,  even  though  this  study  didn’t  prove  that  positive  effect  of  visual  illustration  on  learner’s 
achievement.  However,  it  is  clear  that  using  illustration  in  multimedia  instructional  material  increases  the 
learning  interest  of  learner.  Learner’  interest  is  very  complex  psychological  construct  (Krapp,  Hidi  & 

Renninger,  1992).  Instructional  designer  or  educator  need  to  consider  the  potential  benefits  of  using  different 
types  of  illustration  when  they  develop  multimedia  based  instructional  material. 
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Designing  and  Development  of  EPSS  (Electronic  Performance  Support  System):  Case 
of  (CRMT)  Course  Resource  Management  Tool 
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Florida  State  University 

Introduction 

Within  the  realm  of  the  knowledge-based  economy,  technology  is  considered  as  one  of  the  main  force 
for  the  contemporary  organizational  developments.  EPSS  provides  solution  when  there  are  specific  needs  for 
performing  job  task  effectively.  Generally,  it  provides  help,  demonstrations,  advice,  customized  templates, 
access  to  database,  or  any  other  support  that  the  performer  needs  to  perform  a task  or  series  of  tasks  (Brown, 
1996;  Wager,  2002).  EPSS  does  not  only  refer  to  individual  performance  support  system  but  also  encompass 
organizational  performance  support  system.  This  is  the  reason  why  the  EPSS  should  be  tightly  linked  to  the 
integral  part  of  the  task.  Starting  from  this  point  of  view,  Course  Resource  Management  Tool  (CRMT)  was 
designed  and  developed  to  support  staff  member’s  performance  in  managing  course  resource. 

The  design  and  development  of  Course  Resource  Management  Tool  (CRMT) 

The  performance  setting  describes  a learning  resource  center  in  college  of  education  in  southeastern 
area  university.  The  resource  center  serves  the  staff  and  students  in  college  of  education.  The  functions  of  the 
department  resource  center  include  managing  books,  journals,  CDs/CD  ROMs,  videotapes  and  teaching 
equipment.  This  resource  center’s  staffing  includes  a supervisor  and  five  graduate  assistants  (GA)  to  run  the 
operation.  The  resource  center  has  to  meet  the  needs  of  approximately  thirty  teaching  staff  and  two  hundred 
students . 

The  resource  center  had  problems  monitoring  the  resources  loaned  out  to  students.  The  current  system 
of  loaning  resources  is  still  the  traditional  pen  and  paper  recording.  The  center  is  also  facing  tedious  work  in  the 
booking  of  teaching  equipment.  There  is  no  system  in  managing  the  booking  of  equipment  resulting  in  huge 
time  consumption  and  plenty  paperwork  to  be  done  by  GA(Graduate  Assistants.  In  order  to  minimize  the 
unnecessary  task  process  and  maximize  the  performance  of  staff  members,  CRMT  was  designed  and  developed. 
The  purposes  of  CRMT  are  (1)  To  monitor  all  resources  and  equipment  using  in  an  efficient  and  effective 
electronic  system  (Annual  inventory  check  and  loaning  of  equipment  and  resources),  (2)  To  ensure  all  resources 
to  be  accountable  by  linking  with  the  registrar.  (3)  To  generate  reports  on  the  frequency  of  resources/equipment 
for  future  budgeting,  planning  and  procurement,  (4)  To  generate  sign  in  / sign  out  forms  for  TA  (Teaching 
Assistants,  and  (5)  To  monitor  servicing  schedule  for  all  equipment.  The  work- flow  analysis  was  broken  into 
three  phases  for  analysis,  pre -semester,  during  semester  and  post-semester.  Since  GAs  are  main  staff  members 
of  the  center,  the  work  flow  was  described  based  on  the  main  tasks  they  have  to  carry  out  before  the  semester, 
during  the  semester,  and  after  the  semester.  The  Microsoft  Access  program  was  selected  as  a development  tool. 
During  development  phase,  input  and  output  forms  and  reports  were  designed  along  with  detailed  database  table 
structure.  The  formative  evaluations  were  conducted  separately  with  one  week  after  the  other.  After  the  first 
evaluation,  researcher  did  some  changes  before  conducting  the  second  evaluation.  They  were  given  the  user’s 
guide  and  the  EPSS  program,  and  they  were  told  to  open  and  run  the  database  at  their  own  time  and  pace 
without  any  intervention  from  us.  All  inputs  (both  verbal  and  non-verbal)  were  taken  and  changes  were  made  on 
those  deemed  necessary,  the  amended  inputs  were  consolidated  and  summarized  in  a table.  Table  1 shows  the 
suggestions  made  and  their  corresponding  changes.  Based  on  the  suggested  evaluation  results,  previous  version 
has  been  revised  and  finalized.  Finally,  user  manual  and  final  development  report  were  documented.  The  target 
level  of  this  EPSS  was  work  level.  For  group  level,  EPSS  refers  to  the  electronic  system  that  provides  integrated 
access  to  information,  advice,  learning  experiences,  and  tools.  The  final  object  of  group  level  of  EPSS  is  to  help 
someone  perform  a task  with  cooperation  and  support  from  other  people.  This  presentation  will  provide  detailed 
process  of  design  and  development  of  work  level  EPSS  with  tangible  examples.  Therefore,  it  will  be  a good 
case  for  the  EPSS  designer,  especially  who  have  been  interested  in  designing  work  level  EPSS. 

Performance  Opportunity 

Currently,  the  resource  center  has  problems  monitoring  the  resources  loaned  out  to  students.  The 
current  system  of  loaning  resources  is  still  the  traditional  pen  and  paper  recording.  The  center  is  also  facing 
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tedious  work  in  the  booking  of  teaching  equipment.  There  is  no  system  in  managing  the  booking  of  equipment 
resulting  in  huge  time  consumption  and  plenty  paperwork  to  be  done  by  GAs. 

Work -flow  Analysis 

The  new  work- flow  process  is  broken  into  three  phases  for  analysis.  The  three  phases  are  pre -semester, 
during  semester  and  post-semester. 

Pre-semester  phase 

GAs  will  receive  course  and  students’  information  from  registry  office  prior  to  the  commencement  of 
semester.  The  information  includes  courses  that  EPLS  offers  for  that  semester,  details  of  courses’  information 
and  details  of  registered  students’  information.  This  new  Microsoft  Access  program  will  require  some  system 
integration  with  existing  registrar’s  database  to  obtain  systems  compatibility.  The  information  obtained  from 
registrar  will  update  the  new  database.  Before  a new  semester  begins,  GAs  will  send  email  to  the  teaching  staff 
of  EPLS  to  remind  them  to  submit  their  resources  and  equipment  requirement  for  their  teaching  needs  for  the 
new  semester.  The  email  will  include  attachment  of  resources  and  equipment  request  form  and  faculty 
authorization  form.  Upon  the  receipt  of  resource  and  equipment  (books,  journals,  CDs,  videotapes,  and 
equipment)  requests  from  teaching  staff,  GAs  will  do  an  inventory  check  and  ensure  all  resources  and 
equipment  are  available  and  proceed  to  update  the  database.  GAs  will  generate  a sign  in  / sign  out  file  for  all 
teaching  assistants  to  check  out  the  equipment.  They  will  generate  lists  of  existing  resources  and  equipment  for 
documentation  purposes.  In  addition,  GAs  will  generate  an  equipment  schedule.  This  schedule  will  allow  GAs, 
who  are  on  duty  to  prepare  the  equipment  for  TAs. 

During  semester  phase 

The  similar  process  mentioned  earlier  will  also  take  place  for  ad-hoc  and  last  minute  request  from 
teaching  staff  during  the  semester.  The  primary  tasks  that  GAs  are  responsible  are  transaction  of  resources,  and 
booking  and  signing  in/out  of  equipment. 

Transaction  of  Resources  Students  who  are  currently  registered  for  courses  can  check  in/check  out 
required  book,  journals,  CDs  and  videotapes.  GAs  will  request  student’s  ID  to  check  if  there  is  any  outstanding 
fine.  They  will  also  check  the  availability  of  requested  resources  using  the  database.  If  resource  is  available  and 
there  is  no  outstanding  fine,  GAs  will  proceed  to  check  in  or  out  the  resources  required  by  students.  GAs  will 
print  the  receipt  of  the  transaction  for  the  student.  The  transaction  is  recorded  in  the  database.  When  the 
resource  is  returned,  GAs  check  in  the  resource  and  collect  fine  if  there  is  any  from  the  student.  GAs  will  place 
the  resources  back  to  the  bookshelf. 

Booking  and  Signing  In/Out  of  Equipment  During  the  semester,  ad-hoc  requests  for  equipment  are 
likely  to  occur.  GAs  will  check  available  equipment  before  confirming  the  ad-hoc  requests.  GAs  will  prepare 
equipment  for  TA’s  collection  on  the  required  day  and  time.  GAs  will  also  ensure  all  items  of  equipment  are 
accountable  after  TAs  return  the  equipment;  this  includes  proper  handing  and  taking  over  of  equipment  through 
proper  documentation. 

Post-semester  phase 

GAs  have  the  following  responsibilities: 

> Ensure  all  resources  are  checked  in. 

> For  those  overdue  resources,  GAs  will  contact  the  affected  students  and  remind  them  to  pay  up  by  a 

specific  date. 

> GAs  will  send  an  email  to  registrar  with  a list  of  students  who  have  not  paid  up  their  outstanding  fines. 

Registrar  will  not  allow  students  to  register  for  class  until  they  have  paid  up. 

> Ensure  all  equipment  are  checked  in  and  inform  TAs  who  have  not  returned  the  equipment. 

> Generate  report  of  overdue  payment  for  accounting  purposes 

> Save  information  into  different  file,  for  example  Fall  2005 

> Arrange  for  equipment  to  be  sent  for  service  and  repair. 

Inputs 

> Course  Information: 
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Course  Code,  Course  Title,  Course  Instructor,  Course  Day,  Course  Time,  Course  Room  & 
Course  TA 

> Enrollment  Information  (from  Registrar): 

Enrollment  ID,  Student  ID  & Course  Code 

> Equipment  Information: 

Equipment  ID,  Equipment  Type,  Purchased  Year,  Estimated  Shell-Life,  Servicing  Schedule, 
Servicing  Company  & Company  Contact  No 

> Equipment  Booking  Information: 

Booking  ID,  Course  Code,  Equipment  ID  & Equipment  Status 

> Resource  Information: 

Resource  ID,  Course  Code,  Resource  Type,  Resource  Title,  Resource  Year,  Resource  Author, 
Resource  Status  & Resource  Paid 

> Student  Information: 

Student  ID,  Student  Name,  Semester,  Student  Address,  Student  Major,  Student  Phone  No.  & 
Student  Email. 

> Transaction  Information: 

Transaction  ID,  Resource  ID,  Student  ID,  Date  Out,  Date  Due  & Check  in  Date 

Outputs 

The  output  (in  the  form  of  a report  or  form)  should  reflect  the  above  information  in  addition  to  the 
following: 

Forms 

> Check  Out 

> Check  In 

> Add/Edit  Resource  Information 

> Add/Edit  Course  Information 

> Booking  of  Equipment 

> Edit  Existing  Bookings 

> Add/Edit  Equipment  List 

> Sign  In/Out 

Reports 

> Course  Resource  Report 

> Overdue  Resource  Report 

> Annual  Resource  List 

> Overdue  Payment  Report 

> Annual  Equipment  List 

> Servicing  Schedules 


Data  Tables 

The  data  tables  include  Course,  Enrollment  (From  registrar).  Equipment,  Equipment  Booking, 
Resource,  Student  and  Transaction.  The  fields  of  each  table  are  described  below. 


Course  table  and  its  fields 


Field 

Data  Type 

Example 

Notes 

Course  Code 

Text 

EME  6613 

Primary  Key 

Course  Title 

Text 

Electronic  Performance 
Support  System 

Course  Instructor 

Text 

Walter  Wager 

Course  Day 

Text 

Thu 

Course  Time 

Text 

12:30-15:15 

Course  Room 

Text 

126 

Course  TA 

Text 

Eileen 
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Enrollment  table  and  its  fields 


Field 

Data  Type 

Example 

Notes 

Enrollment  ID 

Text 

Primary  Key 

Student  ID 

Text 

Foreign  Key 

Course  Code 

Text 

Foreign  Key 

Equipment  table  and  its  fields 


Field 

Data  Type 

Example 

Notes 

Equipment  ID 

Text 

DC0001 

Primary  Key 

Equipment  Type 

Text 

Digital  Camera 

Purchased  year 

Date/Time 

3/30/20001 

Estimated  Shelf-Life 

Text 

5 

Servicing  Schedule 

Text 

Once  a year 

Servicing  Company 

Text 

SONY 

Company  Contact  No 

Text 

8500001234 

Equipment  Booking  table  and  its  fields 


Field 

Data  Type 

Example 

Notes 

Booking  ID 

AutoNumber 

1 

Primary  Key 

Course  Code 

Text 

EDF  5400 

Foreign  Key 

Equipment  ID 

Text 

DC0001 

Foreign  Key 

Equipment  Status 

Yes/No 

0 

Resource  table  and  its  fields 


Field 

Data  Type 

Example 

Resource  ID 

Text 

BK0001 

Course  Code 

Text 

EME5601 

Resource  Type 

Text 

Book 

Resource  Title 

Text 

Introduction  to 
fnstmctional  Systems 

Resource  Year 

Text 

1996 

Resource  Author 

Text 

Kanazas  & Rothwell 

Resource  Status 

Yes/No 

0 

Resource  Paid 

Yes/No 

□ 

Notes 


Primary  Key 


Foreign  Key 


Student  table  and  its  fields 


Field 

Data  Type 

Example 

Student  ID 

Text 

9000 

Student  Name 

Text 

SangHoon  Park 

Semester 

Text 

Fall  02 

Student  Address 

Text 

2321  Continental  Ave 
#120  Tallahassee  FL 
32304 

Student  Major 

Text 

IS 

Student  Phone  No. 

Text 

8502222223 

Student  Email 

Text 

Psh_fsu@hotmail.com 

Notes 


Primary  Key 


Transaction  table  and  its  fields 


699 


Field 

Data  Type 

Example 

Notes 

Transaction  ID 

Auto  Number 

1 

Primary  Key 

Resource  ID 

Text 

VI0001 

Foreign  Key 

Student  ID 

Text 

9010 

Foreign  Key 

Date  Out 

Date/Time 

10/27/2002 

Date  Due 

Date/Time 

11/18/2002 

Check  in  Date 

Date/Time 

12/24/2002 

Formative  Evaluation 

Method 

The  formative  product  evaluation  is  the  appraisal  of  instructional  sequences  and  materials  during  their 
stage  of  formulation  and  development.  The  major  purpose  of  formative  product  evaluation  is  to  provide  both 
descriptive  and  judgmental  information  regarding  the  worthiness  of  an  instructional  experience  (Rothwell  and 
Kazanas,  1992).  In  this  paper,  since  the  purpose  of  this  tool  was  performance  improvement,  formative 
evaluation  was  conducted  in  terms  of  performance  experience.  Among  the  four  major  approaches  of  formative 
evaluation,  individualized  pretests  and  pilot  tests  were  conducted.  Two  potential  users  from  the  learning 
resource  center  offered  assistance  to  be  involved  in  our  formative  evaluations. 

Result 

The  formative  evaluations  were  conducted  separately  with  one  week  after  the  other.  After  the  first 
evaluation,  researcher  did  some  changes  before  conducting  the  second  evaluation.  They  were  given  the  user’s 
guide  and  the  EPSS  program,  and  they  were  told  to  open  and  run  the  database  at  their  own  time  and  pace 
without  any  intervention  from  us.  All  inputs  (both  verbal  and  non-verbal)  were  taken  and  changes  were  made  on 
those  deemed  necessary,  the  amended  inputs  were  consolidated  and  summarized  in  a following  table.. 


S/No 

Suggestions 

Changes 

01 

Program  should  be  placed  on  desktop 

Resource  and  equipment  tool  has  been 
placed  on  desktop,  on  clicking  the  icon, 
the  main  switchboard  will  appear 

02 

In  all  edit/add  functions,  program  should  have  a FIND 
button  to  search  for  desired  items 

Previously,  our  program  has  only  some 
functions  that  have  FIND  button,  after 
the  change,  most  functions  have  FIND 
button 

03 

To  add  new  record,  users  who  have  no  knowledge  on 
Microsoft  Access  do  not  know  how  to  do  it. 

User  guide  has  amended  to  include  what 
to  do,  where  to  find  the  button  and  also  a 
picture  of  the  button  to  facilitate  users’ 
learning 

04 

Functions  should  be  arranged  accordingly  to  the  frequent 
of  use 

The  Check  Out  and  Check  In  functions 
have  been  placed  above  Edit/ Add  Course 
and  Resource  Information 

05 

Users  did  not  understand  the  significance  of  Yes  and  No 
in  the  status  column  in  the  Booking  of  Equipment 
function 

A short  message  is  placed  beside  the 
column  explaining  its  significance. 
Amendments  were  also  made  to  the 
user’s  guide 

06 

Users  did  not  like  the  PK  and  FK  on  some  of  the  field 
titles 

All  PK  and  FK  were  removed  from  the 
field  titles 

07 

Title  headings  should  be  placed  on  several  sub-forms,  for 
example  Booking  of  Equipment  function 

All  title  headings  have  been  amended  so 
that  users  know  what  they  represent 

08 

Users  suggested  more  prompt  and  short  messages  to 
remind  the  steps  involved  in  a particular  function 

1.  Steps  have  been  included  in  Check  Out 
function  to  assist  users 

2.  Short  messages  have  also  been 
included  in  all  functions  to  remind  users 
what  to  do.  For  example,  in  the  booking 
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of  equipment,  a short  message  of  “Click 
Update  to  check  whether  booking  is 
confirmed”  is  added 

09 

Users  suggested  using  pictures  for  equipment 

Pictures  are  inserted  for  each  equipment 

10 

Users  suggested  the  layout  to  be  more  attractive,  they 
found  them  too  dull  and  boring 

Proper  formatting  and  variety  of  colors 
are  added  to  the  EPSS  to  make  it  more 
aesthetically  appealing 

11 

Users  suggested  that  for  Check  Out,  it  would  be  good  to 
remind  users  to  type  in  the  necessary  information 

Different  colors  and  font  are  used  to 
distinguish  the  entry  box  to  key  in  the 
data.  A brief  statement  is  used  too 
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Abstract 

This  paper  describes  the  experiences  of  three  middle  school  teachers  following  a two -week  summer 
workshop  in  which  they  were  introduced  to  a technology-enhanced  problem-based  learning  ( PBL ) pedagogy. 
Based  on  their  collaborative  experiences  during  the  school  year  developing  and  implementing  a PBL  unit,  the 
three  teachers  increased  their  confidence  in  using  technology  and  indicated  shifts  in  their  pedagogical  beliefs 
regarding  classroom  instruction.  Results  suggest  that  administrators’  continuous  support  and  collaboration 
with  other  teachers  were  keys  to  teachers’  successful  implementation  of  technology-enhanced  PBL  units. 

Introduction 

In  recent  decades,  teachers,  instructional  designers,  and  other  educators  have  increasingly  been  urged 
to  adopt  a variety  of  constructivist  approaches  in  order  to  facilitate  student-centered  learning  environments 
(Becker,  2000:  Howard,  McGee,  Schwartz,  & Purcell,  2000).  A particular  emphasis  of  this  movement  has 
shifted  the  focus  from  teacher  to  learner,  inviting  learners  to  take  active  roles  in  their  learning  (Means,  1995; 
Reigeluth,  1999).  Among  various  constructivist  approaches,  problem-based  learning  (PBL)  has  been  advocated 
as  an  exemplar  because  it  promotes  students’  understanding,  integration,  and  retention  of  concepts,  facts,  and 
skills  (Gallagher,  1997;  Savery  & Duffy,  1995). 

A PBL  learning  approach  is  based  on  the  use  of  ill-structured  problem  situations  that  are  complex, 
requiring  students  to  develop  expertise  in  information  seeking  and  making  decision  to  solve  the  problem. 
Because  the  problem  situations  are  messy,  confusing,  and  complex,  students  need  to  gather  information  in  order 
understand,  define,  and  solve  the  problems.  During  an  authentic  problem  solving  process,  students  are  able  to 
develop  their  own  approaches  and  set  their  own  goals.  Under  the  guidance  and  coaching  of  a skillful  teacher, 
students  work  collaboratively  to  inquire,  investigate,  and  plan  their  activities  (Sage,  2000). 

This  paper  describes  the  experiences  of  three  middle  school  teachers  as  they  implemented  a 
technology -enhanced  problem-based  learning  (PBL)  approach  in  their  social  studies  curricula  for  the  first  time. 
We  describe  the  challenges  they  faced  as  well  as  the  factors  that  they  perceived  enabled  them  to  be  successful. 
Furthermore,  we  describe  various  strategies  they  need  in  the  professional  development  for  the  technology 
integration  at  the  different  stage. 


Theoretical  Framework 

Technology  changes  have  resulted  in  the  rapid  increase  of  computer  usage  in  schools.  According  to  the 
National  Center  for  Education  Statistics  (NCES,  2000),  nearly  all  public  school  teachers  (99  percent)  reported 
having  computers  in  their  schools.  Internet  connectivity  in  K-12  classrooms  increased  from  65  percent  in  1995 
to  95  percent  in  1999  (Web-Based  Education  Commission,  2000).  However,  NCES  reported  that  nearly  70 
percent  of  teachers  still  didn’t  feel  well-prepared  to  use  computers  and  the  Internet  in  their  teaching.  Teachers’ 
preparation  and  training  to  use  technology  is  a key  factor  to  consider  when  examining  their  instructional  use  of 
computers  and  the  Internet. 

In  1998,  Lewis  listed  a number  of  barriers  to  effective  professional  development  including 
opportunities  to  practice,  access  to  outside  resources  and  expertise,  and  support  from  the  community,  and 
emphasized  the  importance  of  having  on-site  assistance  and  support  while  teachers  attempt  to  develop  and 
implement  new  instructional  practices.  According  to  Trotter  (1999),  teachers  who  received  instruction  related  to 
both  technology  skills  and  technology  integration  ideas  felt  significantly  more  prepared  to  use  technology  in 
their  teaching  compared  to  teachers  who  received  instruction  of  just  one  type. 
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Sage  (2000)  contended  that  a problem-based  learning  (PBL)  approach  was  an  effective  way  to 
integrate  technology  into  the  classroom.  She  defined  PBL  as  “experiential  learning,  organized  around  the 
investigation  and  resolution  of  messy,  real-world  problems’'  (p.  7).  Also,  Hill  (1999)  suggested  that  teacher 
technology  development  can  be  based  on  the  same  problem-centered  methods  that  are  suggested  for  students  in 
problem-based  learning.  Because  technology  is  a critical  tool  for  information  searching,  modeling  task  or 
content  decision  making,  and  presenting  solutions  during  PBL  activities,  technology  integration  with  PBL  can 
be  a meaningful  learning  experience  for  both  teachers  and  students  (Jonassen,  Howland,  Moore,  & Marra, 

2003). 

Although  some  literature  is  available  regarding  the  benefits  of  staff  development  focused  on  promoting 
a technology-enhanced  PBL  approach,  previous  research  has  not  looked  at  how  teachers  adapt  their  classroom 
practices  to  implement  the  suggested  strategies  over  an  extended  period  of  time. 

Purpose 

In  this  case  study  we  examined  the  experiences  of  three  teachers’  at  three  different  times  during  the 
year  following  a 2-week  technology  integration  summer  workshop  involving  enhanced  problem-based  learning 
(PBL).  Specifically,  the  research  questions  guiding  data  collection  and  analysis  included: 

♦ What  are  teachers’  perceptions  of  and  pedagogical  beliefs  about  technology-enhanced  PBL? 

♦ What  kinds  of  barriers  and  support  do  teachers  encounter  while  implementing  technology-enhanced 
PBL? 

♦ What  strategies  are  perceived  as  being  most  in  developing  teachers’  ability  to  implement  technology 
enhanced  PBL? 


Methods 

This  study  began  in  July  2002  and  continued  through  June  2003  as  part  of  a Technology  Innovation 
Challenge  Grant,  funded  by  the  U.S.  Department  of  Education.  A 2-week  summer  institute,  focused  on 
technology  enhanced  PBL,  was  provided  to  kick  off  the  staff  development  process. 

Three  teachers  from  the  same  middle  school,  located  in  a small  rural  community  in  the  Midwest, 
participated  and  developed  a PBL  unit  together  during  the  summer  institute.  The  first  participant,  Carrie,  had 
taught  both  science  and  social  studies  in  the  sixth  grade  for  four  years.  The  second  participant,  Jake,  was  in  his 
second  year  of  teaching  sixth  and  seventh  grade  social  studies.  The  third  participant,  David,  was  in  his  third 
year  of  teaching  social  studies  and  reading  in  the  sixth  grade. 

Preliminary  survey  data  were  used  to  assess  the  participants’  computer  skills,  frequency  of  technology 
use  in  the  classroom,  and  teaching  beliefs  and  practices.  Participants  responded  to  55  questions  with  5-point 
Likert  scale  adapted  from  the  Becker’s  survey  (e.g.,  “I  ask  students  to  work  in  a small  group.”  “Students  in  my 
class  pursue  information  related  to  personal  interests.”)  (Becker,  2000).  These  data  were  collected  at  the 
beginning  of  the  summer  workshop,  2002  and  at  the  end  of  the  spring  semester,  2003. 

Qualitative  data  were  collected  from  various  sources  including  teacher  interviews,  field  notes,  and 
teachers’  journals.  The  first  week  of  the  summer  institute  focused  on  an  introduction  to  PBL,  a PBL  modeling 
activity,  and  various  software  applications  (Internet  search  techniques,  web  page  development,  spreadsheets, 
and  an  online  course  management  system  called  ANGEL).  During  the  second  week  of  the  institute  the  teachers 
worked  collaboratively  to  develop  their  own  PBL  unit.  A daily  reflective  journal  was  kept  by  each  teacher  and 
the  first  interview  was  conducted  at  the  end  of  the  workshop.  The  second  interview  was  conducted  in  the  fall, 
2002  semester  and  the  final  interview  in  spring,  2003.  The  researchers  also  observed  classroom  activities  and 
final  student  presentations  that  culminated  the  PBL  unit. 

Results 

Stage  1:  The  Summer  Institute 

The  participants  indicated  that  overall  the  summer  workshop  was  very  beneficial  in  improving  their 
technology  skills  and  knowledge  and  all  three  reported  an  increase  in  confidence  levels  through  hands-on 
activities.  Through  the  PBL  modeling  activity,  which  included  collaborative  activities  completed  with  k- 12 
students,  teachers  gained  insights  into  the  role  of  the  teacher  and  made  connections  with  how  PBL  can  be 
implemented  in  their  classrooms.  Before  the  workshop  they  indicated  that  they  felt  uncomfortable  using 
technology  in  the  classroom.  One  indicated  that  using  technology  was  a hassle  and  unreliable  in  improving 
student  achievement.  However,  after  the  workshop,  they  all  felt  comfortable  using  a variety  of  software  and 
demonstrated  improved  skills  in  almost  every  area.  Although  the  participants  felt,  overall,  that  the  workshop 
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was  very  beneficial  to  them,  there  were  a few  areas  needing  improvement.  The  teachers  wanted  to  have  more 
examples  and  evidence  of  how  a PBL  unit  can  actually  work  in  their  classrooms.  They  were  concerned  with  the 
reality  of  actually  implementing  a real  unit. 

The  teachers  reported  that  the  collaboration  with  professors  and  colleagues  using  hands-on  activities 
and  development  of  a unit  meeting  their  needs  were  very  positive  features. 

Stage  2:  Changes  Following  the  Summer  Institute 

Following  the  summer  workshop  two  of  the  three  participants  indicated  that  they  were  using 
technology  in  the  classroom  with  much  greater  frequency  and  all  of  them  felt  more  comfortable  with  various 
software  applications. 

The  researchers  hoped  that  the  participants  would  have  implemented  their  PBL  units  by  this  interview, 
but  instead  found  that  the  teachers  had  faced  many  barriers.  The  first  barrier  was  losing  their  common  team 
preparation  time.  Because  of  this,  teachers  could  only  communicate  while  passing  each  other  in  the  hallway  or 
by  meeting  before  or  after  school.  Another  barrier  was  the  time  needed  to  prepare  students  for,  and  to  give,  the 
standardized  tests  required  by  the  state  at  the  beginning  of  the  semester.  This  left  teachers  with  little  time  to 
introduce  technology  to  the  students  and  to  practice  mini- PBL  strategies  during  class  time.  Because  of  these 
barriers,  none  of  the  participants  implemented  their  PBL  units  in  the  fall  2002  semester. 

Stage  3:  Changes  Following  Implementation  of  the  PBL  Unit 

All  three  participants  conducted  their  three  week  PBL  units  in  spring  2003,  during  which  time  they 
involved  their  sixth  grade  students  in  questions  related  to  the  history  of  their  community.  Survey  data  collected 
at  this  time  indicated  that  the  teachers  increased  in  their  technology  expertise  ( M=  3.43,  SD=  .31)  from  the 
summer  (M=  3.05  SD=  .30)  and  demonstrated  shifts  in  their  beliefs  about  student  centered  learning  ( M=  3.33, 
SD=  .94)  from  summer  ( M=  3.08,  SD=  1.56)  Student  computer  use  in  the  classroom  also  increased  (M= 2.94, 

SD=  .21)  from  summer  (M=  2.44,  SD=  0.08) 

Based  on  qualitative  data,  teachers  perceived  that  their  levels  of  technology  confidence  and  PBL 
understanding  were  higher  than  before  implementation.  First,  teachers’  technology  comfort  level  was  improved 
through  using  a variety  of  software;  they  reported  only  minor  technical  problems  during  the  PBL  units. 
Furthermore,  the  network  system  was  improved  and  technical  support  personnel  were  very  quick  in  trouble 
shooting  any  problems.  Second,  they  realized  the  role  of  teachers  as  a facilitator  and  students  as  a researcher 
and  instructor  to  other  students  actively  engaged  with  ownership  and  responsibility  in  their  learning. 

Although  teachers  believed  they  had  succeeded  with  the  PBL  unit,  they  experienced  barriers  related  to 
time  and  resources.  Forty-five  minute  class  sessions  were  too  short  for  students  to  use  computers  for 
brainstorming,  locating  information,  discussing  topics,  and  organizing  information.  Teachers  were  required  to 
work  together  after  school  because  of  losing  team  preparation  time.  In  addition,  the  PBL  topic,  focused  on  the 
community’s  history,  made  finding  online  resources  difficult  and  students  had  to  be  more  dependent  on  the 
local  library  and  interviews  with  community  members.  Fortunately,  the  school  district  has  extended  class  time 
to  sixty-five  minutes  and  re -implemented  the  team  preparation  time  for  the  next  school  year.  The  local  library 
is  also  supporting  the  unit  and  adding  student  incentives  by  displaying  the  students’  work  to  the  community. 

Based  on  their  experiences  with  their  PBL  units,  there  was  a distinctive  change  in  teachers’ 
pedagogical  beliefs  pertaining  to  using  technology  enhanced  PBL.  Due  to  lack  of  comfort  and  technical  issues 
experienced  in  the  past,  all  of  the  participants  used  to  think  of  technology  as  a nuisance  unnecessary  for  student 
achievement  and  learning.  However,  teachers  became  confident  using  technology  enhanced  PBL.  They  also 
realized  that  the  students  were  more  actively  engagement  in  learning,  and  students  were  learning  technology 
skills  more  quickly  as  they  helped  each  other. 

Finally,  the  participants  suggested  the  ideal  workshop  format  for  a technology  enhanced  PBL 
workshop  is  one  that  includes  other  teachers  with  different  levels  of  technology  and  PBL  experience.  For 
teachers  at  the  beginning  level,  hands-on  activities  combined  with  developing  their  own  units  alongside  teachers 
with  previous  experiences,  was  perceived  as  most  beneficial.  For  the  intermediate  level,  they  preferred 
receiving  some  practical  guidelines  that  could  refresh  their  knowledge,  new  technology  skills,  more  hands-on 
activities  with  their  own  units  to  modify,  and  feedback  from  other  teachers  outside  of  their  own  groups. 

Discussion  and  Implications 

Through  this  s tudy,  we  found  changes  at  each  stage  of  staff  development  implementation.  In  the  first 
stage  involving  the  PBL  modeling  activity  and  hands-on  activity,  teachers  developed  technology  skills  and 
described  increases  in  feeling  “comfortable”  with  technology.  The  researchers  interpreted  this  to  mean  that  they 
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were  no  longer  scared  of  encountering  technical  problems  involving  software  applications  and  were  also  more 
prepared  to  help  students  use  the  technology  and  implement  it  in  their  classrooms.  However,  one  of  teachers 
expressed  feelings  of  being  overwhelmed,  concerns  about  time  allocation,  collaboration  with  other  teachers  and 
technical  problems.  These  are  significant  barriers  that  must  be  addressed  before  teachers  can  go  back  to  their 
classrooms  and  implement  PBL.  Technical  problems  and  feelings  of  isolation  can  inhibit  teachers  from  ever 
trying  this  different  approach. 

At  stage  two,  we  found  that  two  of  the  three  participants  were  able  to  talk  with  each  other  everyday 
about  teaching  issues,  technology,  and  PBL  because  their  rooms  were  adjacent  to  one  another.  As  a result  of  the 
interview,  these  two  showed  new  changes  in  using  technology  and  strategies  from  PBL  where  the  third  missed 
out  on  the  opportunity  to  participate  in  these  informal  discussions  and  showed  fewer  changes.  This  shows  how 
important  the  team  preparation  time  is  for  collaboration  among  teachers.  Administrative  support  in  areas  like 
scheduling  can  have  a large  impact  on  the  implementation  of  new  teaching  methods. 

The  largest  changes  were  found  in  stage  three,  following  the  implementation  of  the  PBL  units.  All 
three  participants  showed  increases  in  the  frequency  of  technology  use  in  their  classrooms  and  an  increase  in 
comfort,  confidence,  and  shifts  in  pedagogical  beliefs  in  using  technology  enhanced  PBL.  Teachers  adopted 
mini-PBL  activities  with  technology  in  other  units  before  their  three-week  PBL  collaborative  unit.  It  is 
important  to  note  that  the  largest  changes  in  the  areas  of  comfort,  confidence,  and  pedagogical  belief  came  after 
the  teachers  had  actually  experienced  leading  a PBL  unit  in  their  own  classroom  through  collaboration. 

How  can  we  encourage  teachers  to  get  to  the  point  that  they  are  willing  to  implement  PBL  in  their 
classrooms?  Data  from  this  study  suggested  that  effective  staff  development  should  provide  opportunities  for 
teachers  to  practice  with  hands-on  activities  with  the  unit  meeting  teachers’  needs,  and  provide  opportunities  for 
collaboration  with  colleagues  and  experts.  Most  of  all,  continuous  administrator  support  in  providing  team 
preparation  time  and  creating  a school  culture  that  values  the  sharing  of  teachers’  experiences  was  perceived  as 
being  critical  to  the  success  of  teachers’  efforts  to  initiate  change  in  their  classrooms. 
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Human  working  memory  can  be  defined  as  a component  system  responsible  for  the  temporary  storage 
and  manipulation  of  information  related  to  higher  level  cognitive  behaviors,  such  as  understanding  and 
reasoning  (Baddeley,  1992;  Becker  & Morris,  1999).  Working  memory,  while  able  to  manage  a complex  array 
of  cognitive  activities,  presents  with  an  unexpected  peculiarity,  in  that  only  a few  elements  of  information  can 
be  processed  in  working  memory  at  a given  time.  Miller  (1956)  established  that  working  memory  can  only 
maintain  about  seven  elements  of  information  at  a time.  Additionally,  working  memory  under  conditions  where 
rehearsal  is  limited,  may  only  be  able  to  hold  information  active  for  a few  seconds  (Peterson  & Peterson,  1959). 
In  situations  involving  more  complex  cognitive  tasks,  demands  placed  on  working  memory  that  are  not  directly 
related  to  problem  solving  can  hinder  learning  by  exceeding  available  cognitive  resources  (Sweller,  van 
Merrienboer,  & Paas,  1998).  In  such  situations,  instructional  principles  that  avoid  overburdening  working 
memory  and/or  direct  the  learner’s  available  resources  are  needed  to  design  efficient  and  effective  instruction. 

Cognitive  load  theory,  as  conceptualized  by  Sweller  (1988)  and  his  colleagues  in  the  late  eighties,  is 
concerned  with  instructional  design  and  message  design  methods  that  efficiently  manage  the  limited  processing 
capabilities  of  an  individuals  working  memory  while  capitalizing  on  the  extensive  capabilities  of  long  term 
memory  in  order  to  promote  schema  formation  and  improve  intellectual  learning  and  performance  of  complex 
cognitive  tasks. 

Cognitive  load  (CL)  is  defined  as  the  “total  amount  of  mental  energy  imposed  on  working  memory  at 
an  instance  in  time”  (Cooper,  1998,  p.  10).  This  “total”  cognitive  load  is  further  subdivided  into  three 
subcomponents:  intrinsic  cognitive  load  (ICL),  extraneous  cognitive  load  (ECL)  and  germane  cognitive  load 
(GCL)  (Sweller  etal.,  1998). 

Intrinsic  cognitive  load  is  the  load  imposed  on  the  learner  by  the  nature  of  the  instructional  material 
that  must  be  proces  sed  and  learned.  There  is  evidence  to  support  the  indirect  manipulation  of  ICL  by 
incorporating  sequencing  and  layering  strategies  into  the  instructional  design  process  and  learning  task  (Pollock, 
Chandler,  & Sweller,  2002). 

Extraneous  cognitive  load  is  the  load  imposed  by  factors  such  as  instructional  strategies,  message 
design,  interface  design,  and  the  quality  of  instructional  materials  and  learning  environments.  ECL  is  readily 
influenced  by  instructional  design  processes  and  has  been  the  focus  of  much  investigation  (Sweller  et  ah,  1998). 
In  simple  terms,  high  ECL  equates  to  a reduction  in  working  memory  resources  available  for  learning,  while 
low  ECL  equates  to  an  increase  in  working  memory  resources  available  for  learning.  Research  related  to  the 
physical  integration  of  diagrams  and  text  and  the  elimination  of  unnecessary  information  in  order  to  reduce 
demands  on  working  memory  has  been  conducted  with  much  success  in  the  knowledge  domains  of  biology, 
computer-aided  design/computer-aided  manufacturing,  electrical  engineering,  computer  programming  and 
mathematics  (Bobis,  Sweller,  & Cooper,  1993;  Chandler  & Sweller,  1991,  1996;  Kalyuga,  Chandler,  & Sweller, 
1998;  Leung,  Low,  & Sweller,  1997;  Sweller  et  ah,  1998;  Tarmizi  & Sweller,  1988). 

A third  and  final  dimension  of  cognitive  load  is  germane  cognitive  load  and  is  described  as  the  “load 
imposed  by  cognitive  processes  directly  relevant  to  learning”  (van  Merrienboer,  Schuurman,  de  Croock,  & 

Paas,  2002,  p.12).  The  instructional  designer  can  indirectly  manipulate  GCL.  However,  given  that  intrinsic  + 
germane  + extraneous  cognitive  load  equals  total  cognitive  load,  the  combination  of  ECL  and  ICL  must  leave 
sufficient  resources  available  if  GCL  is  to  be  addressed  (Kirschner,  2002;  van  Merrienboer  et  ah,  2002). 

Currently,  the  study  of  cognitive  load  theory  has  provided  hypotheses  and  conclusive  findings 
concerning  appropriate  strategies  for  structuring  instructional  material  as  studied  in  many  technical  knowledge 
domains.  The  focus  of  this  paper  is  to  investigate  the  applicability  of  the  cognitive  load  theory  principles  of 
redundancy  and  split  attention  to  teaching  Manual  Physical  Therapy  Skills. 
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Working  memory  is  loosely  described  as  interconnected  cognitive  mechanisms  that  maintain  newly 
acquired  information  and  retrieve  stored  information  to  an  active  state  for  processing  and  manipulation,  ft  is  in 
working  memory  where  complex  cognitive  tasks  such  as  reasoning  and  problem  solving  occur  (Baddeley,  1992; 
Becker  & Morris,  1999).  ft  is  also  in  working  memory  where  a potential  bottleneck  exists  between  the  learning 
task  at  hand  and  long-term  memory.  Working  memory  while  capable  of  managing  an  impressive  array  of 
cognitive  tasks  is  surprisingly  limited  in  both  capacity  and  duration  (Baddeley,  1992;  Miller,  1956;  Shiffrin  & 
Nosofsky,  1994).  In  contrast,  long-term  memory  effectively  stores  all  of  our  knowledge  (content,  skills  and 
strategies)  on  a permanent  basis,  with  the  ability  to  recall  this  information  being  somewhat  more  variable 
(Baddeley,  1992;  Ericsson  & Kintsch,  1995). 

Element  Complexity  and  Element  Interactivity 

In  situations  with  low-element  interactivity,  such  as  with  serial  processing  tasks,  little  or  no  overlap 
exists  between  elements  but  as  the  number  of  independent  elements  increases,  the  tasks  may  become  difficult. 

In  contrast,  in  situations  with  high-element  interactivity  where  understanding  requires  that  all  elements  be 
maintained  in  working  memory  and  manipulated  simultaneously,  learning  tasks  can  become  exceptionally 
difficult.  In  such  instances,  the  cognitive  load  imposed  by  trying  to  keep  all  elements  in  working  memory  may 
exceed  the  processing  abilities  of  working  memory  (Sweller  & Chandler,  1994). 

Split  Attention  and  Redundancy 

Split  attention  and  redundancy  are  closely  linked  concepts  and  in  many  cases  are  managed  with  similar 
design  principles.  For  example,  if  two  or  more  sources  of  information  cannot  be  understood  in  isolation,  then  a 
split  attention  effect  may  occur  and  if  they  can  be  understood  in  isolation,  a redundancy  effect  may  occur 
(Kalyuga  et  ah,  1998;  Kalyuga,  Chandler,  & Sweller,  1999;  Sweller,  1990,  1994,  1999;  Sweller  & Chandler, 
1994;  Sweller  et  ah,  1998).  Split  attention  and  redundancy  effects  have  been  studied  and  can  be  practically 
categorized  as  (a)  single  format  media  studies,  (b)  dual  format  media  studies  and  (c)  multiple  media  studies 
employing  the  use  of  audio  and  visual  materials.  Last,  important  conditional  factors  such  as  learner  experience 
have  been  identified  across  all  categories. 

First,  the  study  of  split  attention  and  redundancy  effects  across  several  single  format  media  studies 
have  established  consistent  split  attention  and  redundancy  effects,  as  demonstrated  by  significantly  superior 
learner  performance,  improved  test  scores,  decreased  error  rates,  quicker  content  processing  times,  reduced 
completion  times  and  decreased  levels  of  cognitive  load.  In  contrast,  given  the  situated  nature  of  cognitive  load 
research,  specific  levels  of  redundancy,  split  attention  and/or  levels  of  unintelligibility  could  not  be  expressed  as 
formal  prescriptive  principles  that  are  readily  transferable  to  other  learners  and/or  knowledge  domains  (Bobis  et 
al.,  1993;  Chandler  & Sweller,  1991,  1992;  Purnell,  Solman,  & Sweller,  1991;  Tarmizi  & Sweller,  1988). 

Second,  similar  split  attention  and  redundancy  effects  have  been  studied  in  instructional  situations 
involving  two  modes  of  instructional  delivery,  i.e.,  high  complexity  computer  training  with  computer  plus 
manual  approaches  vs.  integrated  manual  or  CBI  only.  Findings  supported  single  media  format  studies  and 
identified  that  instructional  delivery  strategies  that  utilize  two  pieces  of  instructional  material  may  be 
detrimental  to  learning  when  more  complex  knowledge  is  being  learned  (Bobis  et  al.,  1993;  Cerpa,  Chandler,  & 
Sweller,  1996;  Chandler  & Sweller,  1992,  1996;  Purnell  et  al.,  1991). 

Third,  studies  that  incorporate  both  visual  and  auditory  content  have  made  a distinction  between 
principles  related  to  splitting  the  learner’s  attention  between  auditory  and  visual  information  (dual  channel 
coding)  and  those  related  to  split  attention  and  redundancy  effects  in  one  channel  (e.g.,  print)  (Baddeley,  1992; 
Kalyuga  et  al.,  1999;  Reichle,  Carpenter,  & Just,  2000).  In  this  context,  both  auditory  and  visual  information 
may  be  used  to  expand  working  memory,  while  ameliorating  audio-visual  split  attention  effects  (if  any)  and 
further  benefit  form  prior  strategies  that  limit  split  attention  and  redundancy  effects  (Kalyuga  et  al.,  1999; 

Mayer,  Heiser,  & Lonn,  2001;  Mayer  & Moreno,  1998;  Moreno  & Mayer,  1999,  2002;  Tindall-Ford,  Chandler, 
& Sweller,  1997). 

Last,  in  addition  to  the  structure  of  human  memory  and  distinct  dimensions  of  cognitive  load,  learner 
experience  and  the  ability  of  the  instructional  designer  to  correctly  categorize  how  intrinsic,  extraneous  and 
germane  load  will  influence  a given  learner  or  group  has  been  shown  to  have  bearing  on  the  effectiveness  of 
cognitive  load  theory  driven  design  processes.  That  is,  because  a given  instructional  format  is  in  part  dependent 
on  the  experience  of  the  learner,  a format  designed  for  a low  level  learner  may  not  be  suitable  for  a high  level 
learner,  with  the  opposite  also  holding  true  for  a higher  level  learner.  Such  findings  have  provided  support  for 
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the  need  to  design  instruction  and  instructional  processes  to  the  level  of  experience  of  the  learner  and  that 
instruction  that  fails  to  do  so  may  present  with  deleterious  effects  (Kalyuga,  Ayres,  Chandler,  & Sweller,  2003; 
Kalyuga  et  ah,  1998;  Pollock  et  ah,  2002;  Renkl  & Atkinson,  2003;  Yeung,  1999;  Yeung,  Jin,  & Sweller,  1997). 

Cumulatively,  prior  studies  have  provided  powerful  evidence  indicating  that  material  should  typically 
be  presented  without  redundant  features,  and  that  materials  that  cannot  be  understood  in  isolation  should  be 
physically  integrated.  Second,  self-explanatory,  integrated  diagrams  are  presumed  superior  when  redundant  and 
incidental  materials  are  removed.  Third,  split  attention  and  redundancy  effects  are  equally  applicable  to 
multimedia  instructional  modalities  that  incorporate  dual  channel  strategies.  Fourth,  learning  and  transfer  are 
both  favored  by  strategies  that  eliminate  split  attention  and  redundancy  in  technical  areas  and  last,  conditional 
factors  such  as  learner  experience  must  be  accounted  for  within  a given  knowledge  domain. 

Purpose  of  the  study,  Hypotheses  and  Research  Questions 

The  purpose  of  this  study  was  to  test  the  effectiveness  of  realistic  integrated  instructional  materials 
designed  to  control  redundancy  and  split  attention  in  the  teaching  of  complex  Orthopedic  Physical  Therapy 
skills.  Integrated  materials  were  compared  to  non-integrated  materials  and  subjective  ratings  of  cognitive  load, 
post-instruction  written  test  scores,  post-instruction  psychomotor  performance,  and  time  to  complete  specific 
tasks  were  assessed.  The  following  hypotheses  were  tested: 

1.  Participants  who  receive  integrated  instructional  formats  will  achieve  higher  written  posttest  scores  as 
compared  to  control  group  participants  who  receive  non-integrated  instructional  formats. 

2.  Participants  who  receive  integrated  instructional  formats  will  report  lower  subjective  ratings  of  cognitive 
load  as  compared  to  control  group  participants  who  receive  non-integrated  instructional  formats  for  both 
post  instruction  and  post  psychomotor  performance. 

3.  Participants  who  receive  integrated  instructional  formats  will  achieve  higher  performance  scores  on  the 
performance  of  manual  physical  therapy  skills  as  compared  to  control  group  participants  who  receive  non- 
integrated  instructional  formats. 

4.  Participants  who  receive  integrated  instructional  formats  will  have  lower  task  completion  times 
(instructional  unit  and  examination)  as  compared  to  control  group  participants  who  receive  non-integrated 
instructional  formats . 

In  an  attempt  to  identify  the  applicability  of  cognitive  load  theory  constructs  to  a previously 
unexamined  knowledge  domain  (Physical  Therapy)  and  from  the  perspective  of  both  cognitive  and 
psychomotor  performance,  research  questions  that  were  examined  in  this  study  were  as  follows: 

1.  Are  redundancy  and  split  attention  principles  derived  from  Cognitive  Load  Theory  transferable  to  the 
knowledge  domain  of  manual  physical  therapy? 

2.  Will  instructional  materials  designed  in  accordance  with  cognitive  load  theory  design  principles  positively 
influence  learner  attitudes  towards  instruction? 

3.  Will  the  management  of  redundancy  and  split  attention  affect  psychomotor  performance? 

4.  What  aspects  of  the  psychomotor  skills  are  transferred  immediately  following  instruction;  accurately 
without  practice  and  without  feedback? 


Method 


Participants 

Forty-one  graduate  program  physical  therapy  students  were  recruited  on  a voluntary  basis  and 
scheduled  from  within  the  Department  of  Physical  Therapy  at  a large  Midwestern  university  (Integrated 
instruction  N=9,  Nonintegrated  instruction  N=8)  and  from  within  the  Program  in  Physical  Therapy  at  a second, 
but  smaller  Midwestern  university  (Integrated  instruction  N=12,  Nonintegrated  instruction  N=12). 

Materials 

Two  questionnaires,  two  instructional  units  of  equivalent  content,  a psychomotor  performance  rubric,  a 
date/time  log,  protocol  instructions  and  participant  instruction  were  developed  and  reviewed  by  four  instructors 
and  piloted  on  six  advanced  students.  Both  instructional  units  were  based  on  an  actual  curricular  unit  of 
instruction  and  the  content  was  directly  applicable  to  clinical  practice  with  the  exception  of  only  presenting  a 
single  technique  in  the  stimulus  materials. 

Non-integrated  instructional  unit 
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The  non-integrated  instructional  unit  contained  a brief  introduction  and  two  short  knowledge  sections. 
Section  one  provided  a brief  introduction  to  the  principles  of  orthopedic  provocation  and  alleviation  (techniques 
used  to  increase  or  decrease  patient  symptoms  based  on  patient  movement,  positions,  or  manual  contact). 

Section  two  described  the  process  for  performing  a specific  provocation  and  alleviation  test.  Specifically,  the 
procedure  presented  in  the  units  of  instruction  required  a series  of  appropriately  sequenced  steps  that  could  be 
used  to  help  clinically  localize  the  primary  anatomical  region  of  dysfunction  for  an  orthopedic  patient 
complaining  of  generalized  low  back,  pelvic  and  leg  pain  with  weight  shifting  onto  the  painful  extremity  (leg). 

Integrated  instructional  unit 

Sections  and  content  were  equivalent,  though  message  design  attributes  for  the  integrated  instructional 
unit  sought  to  eliminate  all  redundancy  and  split  attention  effects.  Specifically,  content  was  the  same,  though 
message  design  attributes  for  the  integrated  instructional  unit  subscribed  to  cognitive  load  theory  design 
practices,  i.e.,  content  that  was  unintelligible  in  isolation  was  integrated  and  redundant  information  was 
completely  eliminated. 

Instruments 

The  post-instructional  questionnaire  was  used  to  collect  participant  reported  educational  and 
biographical  data  (age,  gender,  GPA,  and  prior  academic  degrees),  and  a subjective  rating  of  the  difficulty  of 
the  instructional  materials  based  on  the  prior  literature  (Kalyuga,  Chandler,  & Sweller,  2000;  Paas,  E.  Tuovinen, 
Tabbers,  & van  Gerven,  2003).  Finally,  the  questionnaire  asked  the  participant  to  rate  attitudes  towards 
learning  (quality,  effectiveness,  relevance  and  confidence).  Additionally,  the  post-psychomotor  task 
instructional  performance  questionnaire  asked  the  participant  to  rate  the  difficulty  of  the  psychomotor  task 
performance  using  the  same  scale.  The  delayed  written-post  test  consisted  of  18  questions  designed  to  assess 
specific  content  features  of  the  instructional  units  and  the  posttest  psychomotor  rubric  was  used  to  evaluate  the 
psychomotor  performance  of  each  participant. 


Procedure 

All  participants  were  randomly  assigned  to  one  of  two  treatment  groups.  In  all,  41  participants 
successfully  completed  the  treatments  across  1 1 data  collection  sessions  across  1 1 data  collection  sessions 
(average  number  of  participants  per  group  M=3.7).  After  the  written  posttest,  participants  were  escorted  to  a 
separate  laboratory  designated  for  the  psychomotor  assessment,  which  prohibited  participants  from  overhearing 
other  participants  as  they  completed  the  psychomotor  assessment. 

Results 

The  four  hypotheses  were  evaluated  using  a multivariate  analysis  of  variance  and  the  significance  level 
was  set  at  alpha  = .05.  Additionally,  no  consequential  violations  of  normality  and  homogeneity  of  variance 
were  observed.  Educational  and  biographical  data  were  analyzed  and  no  significant  main  effect  was  found;  the 
data  sets  from  the  two  institutions  were  pooled  for  the  analyses.  The  MANOVA  yielded  an  overall  significant 
difference  (Omnibus  F)  between  the  integrated  and  non-integrated  groups,  Pillai’s  Trace:  F(6,  34)  = 6.213,  p < 
.001,  ES  = .52.  Descriptive  statistics  for  both  groups  are  presented  in  Table  1. 

A main  effect  was  found  for  written  posttest  scores,  F(  1 ,39)  = 16.564,  p < .001,  MSt  = 2.12,  ES  = .30. 
Participants  who  received  the  integrated  instructional  format  achieved  significantly  higher  written  posttest 
scores  (M  = 16.00)  as  compared  to  control  group  participants  who  receive  the  non-integrated  instructional 
format  (M  = 14.15)  as  predicted  by  the  first  hypothesis. 

Table  1.  Descriptive  statistics  for  both  groups:  primary  hypotheses. 

Instructional  Format 

Dependent  Variables  Non-integrated  Integrated 


N 20  21 


Exam  total  score 1 

M 

14.15 

16.00 

SD 

1.76 

1.10 

Cognitive  load  rating  one  - instruction2 

M 

3.35 

2.71 

SD 

0.81 

0.85 

Cognitive  load  rating  two  - technique2 

M 

3.50 

2.62 
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SD 

0.95 

1.07 

Psychomotor  rubric  total  score 3 

M 

30.85 

39.76 

SD 

6.05 

4.44 

Time  to  complete  instructional  unit4 

M 

20.15 

19.71 

SD 

6.13 

6.55 

Time  to  complete  written  examination4 

M 

16.95 

18.81 

SD 

3.14 

3.56 

Notes 

1.  Possible  range  for  exam  score  (0-18) 

2.  Possible  range  for  cognitive  load  ratings  one  and  two  ( 1 -7) 

3.  Possible  range  for  psychomotor  rubric  score  (0-44) 

4.  Possible  range  for  instruction  and  written  examination  times  (0-30  minutes) 

Additional  analysis  was  conducted  to  assess  content  structure  scores  as  identified  by  low  element 
interactivity  and  complexity  or  high-element  interactivity  and  complexity  in  the  following  categories:  (a) 
identical  content  (low  element  interactivity  and  complexity),  (b)  redundant  content  in  the  non -integrated  unit  of 
instruction  (high-element  interactivity  and  complexity)  and  (c)  split  attention  content  in  the  non-integrated  unit 
of  instmction  (high  element  interactivity  and  complexity). 

Univariate  analyses  of  content  structure  scores  revealed  that  the  integrated  instruction  group  (M  = 

5.71)  scored  significantly  higher  than  the  non-integrated  instruction  group  (M  = 5.20)  on  the  written  posttest 
when  the  content  was  not  presented  with  redundant  format,  F(  1 ,39)  = 6.82,  p = .013,  MSe  =.34,  ES  = .15,  and 
when  the  content  did  not  have  split  attention  features  (M  = 4.81),  F(  1 ,39)  = 9.73,  p = .003,  MSj,  = .97,  ES  = .20; 
as  compared  to  the  non-integrated  instruction  group  (M  = 3.85).  No  differences  were  noted  with  identical 
presentation  formats  between  groups,  observed  power  value  = .33  (see  descriptive  statistics,  Table  2). 


Table  2.  Descriptive  statistics  for  exam  scores  as  a function  of  content  structure. 


Content  Structure 

N 

Non-integrated 

20 

Instructional  Format 
Integrated 
21 

Identical  content1 

M 

5.10 

5.48 

SD 

0.79 

0.75 

Redundant  content2 

M 

5.20 

5.71 

SD 

0.77 

0.46 

Split  attention  content3 

M 

3.85 

4.81 

SD 

1.18 

0.75 

Notes 

Possible  range  for  exam  scores  in  each  category  (0-6) 

1.  Content  was  identical  in  both  treatments  (no  redundant  or  split  attention  features) 

2.  Content  contained  redundant  information  in  the  non-integrated  treatment 

3.  Content  was  unintelligible  without  mental  integration  in  the  non-integrated  treatment 

A main  effect  was  found  for  subjective  ratings  of  cognitive  load  measured  after  the  completion  of  the 
instruction  F(l,39)  = 6.02,  p = .019,  MSe  = .69,  ES  = .13,  and  after  the  completion  of  the  psychomotor 
performance  task  F(  1,39)  = 7.76,  p = .008,  MS,  = 1.02,  ES  = .17  supporting  hypothesis  two.  Participants  who 
received  the  integrated  instruction  format  reported  significantly  lower  subjective  ratings  of  cognitive  load 
measured  after  instruction  (M  = 2.71),  as  compared  to  control  group  participants  who  received  the  non- 
integrated  instruction  format  (M  = 3.35),  as  predicted.  Additionally,  participants  who  received  the  integrated 
instruction  format  reported  significantly  lower  subjective  ratings  of  cognitive  load  measured  after  psycho  motor 
performance  (M  = 2.62),  as  compared  to  control  group  participants  who  received  the  non-integrated  instruction 
format  (M  = 3.50). 

A main  effect  was  found  for  psychomotor  performance  scores  F(l,39)  = 29.15,  p < .001,  MSe  = 27.90, 
ES  = .43.  As  predicted  by  hypothesis  three,  participants  who  received  the  integrated  instruction  format 
achieved  significantly  higher  rubric  scores  on  the  performance  of  manual  physical  therapy  skills  (M  = 39.76),  as 
compared  to  control  group  participants  who  received  the  non-integrated  instruction  format  (M  = 30.85). 

The  psychomotor  rubric  was  broken  into  three  distinct  sections:  evaluation,  application  and 
comprehension.  Univariate  analyses  of  psychomotor  performance  scores  revealed  that  the  integrated  instruction 
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group  scored  significantly  higher  on  the  evaluation  section  (M  = 7.10):  F(l,39)  = 20.23,  p < .001,  MSe  = 8.91, 
ES  = .34;  and  the  application  section  (M  = 28.67):  F(l,39)  = 13.95,  p < .001,  MSe  = 13.37,  ES  = .26;  of  the 
rubric  as  compared  to  control  group  participants  who  received  the  non-integrated  instruction  format  (M  = 2.90 
and  24.40  respectively).  No  significant  differences  were  noted  on  the  comprehension  section  between  groups 
(observed  power  value  = .37).  Descriptive  statistics  for  evaluation,  application  and  comprehension  rubric 
scores  are  presented  in  Table  3. 


Table  3.  Descriptive  statistics  for  the  psychomotor  rubric:  evaluation,  application  and  comprehension. 


Rubric  Sections 

N 

Instructional  Format 

Non-integrated 

20 

Integrated 

21 

Evaluation1 

M 

2.90 

7.10 

SD 

2.45 

3.43 

Application2 

M 

24.40 

28.67 

SD 

4.88 

1.85 

Comprehension3 

M 

3.55 

4.00 

SD 

1.23 

0.00 

Notes 

1.  Possible  range  for  scores  (0-10) 

2.  Possible  range  for  scores  (0-30) 

3.  Possible  range  for  scores  (0-4) 

Hypothesis  four  stated  that  “Participants  who  receive  integrated  instructional  formats  will  have  lower 
task  completion  times  (instructional  unit  and  examination)  as  compared  to  control  group  participants  who 
received  non-integrated  instructional  formats”.  There  were  no  significant  differences  between  the  groups  on 
time  spent  to  complete  the  instructional  unit  or  the  written  examination.  These  two  variables  had  observed 
power  values  of  .06  and  .41,  respectively. 

In  an  attempt  to  identify  the  applicability  of  cognitive  load  theory  constructs  to  teaching  manual 
physical  therapy  skills,  four  research  questions  were  examined  in  this  study.  The  first  research  question  sought 
to  determine  if  redundancy  and  split  attention  principles  derived  from  cognitive  load  theory  were  transferable  to 
the  teaching  of  manual  physical  therapy  skills.  Statistically  significant  results  in  the  expected  direction  for  three 
of  the  four  hypotheses  provided  support  for  the  transferability  of  cognitive  load  theory  design  principles  to  the 
knowledge  domain  of  manual  physical  therapy  in  this  study.  Specifically,  the  integrated  instruction  group 
achieved  significantly  higher  written  exam  scores  (M  = 16.00):  F(  1,39)  = 16.564,  p < .001,  MSt  = 2.12,  ES  = 
.30,  as  compared  to  the  non-integrated  instruction  group  (M  = 14.15).  Additionally,  the  integrated  instruction 
group  reported  significantly  lower  subjective  ratings  of  cognitive  load  after  the  completion  of  the  instruction 
F(l,39)  = 6.02,  p = .019,  MSe  = .69,  ES  = .13,  (M  = 2.71  and  M = 3.35,  respectively)  and  after  the  completion 
of  the  psychomotor  performance  task  F(  1,39)  = 7.76,  p = .008,  MSe  = 1.02,  ES  = .17,  (M  = 2.62  and  M = 3.50, 
respectively).  Finally,  the  integrated  instruction  group  (M  = 39.76)  achieved  significantly  higher  psychomotor 
performance  scores  F(  1 ,39)  = 29. 15,  p < .001,  MSe  = 27.90,  ES  = .43,  as  compared  to  control  group  participants 
who  received  the  non-integrated  instruction  format  (M  = 30.85). 

The  second  research  question  sought  to  discover  if  instructional  materials  designed  in  accordance  with 
cognitive  load  theory  design  principles  would  positively  influence  learner  attitudes  towards  instruction.  The 
integrated  instruction  group  reported  lower  subjective  ratings  of  cognitive  load  for  the  written  post  test  (M  = 

2.7 1):  F(  1 ,39)  = 6.02,  p = .019,  MSe  = .69,  ES  = .13  and  on  psychomotor  performance  (M  = 2.62),  as  compared 
to  the  non-integrated  instruction  group  (M  = 3.35  and  3.50  respectively).  In  contrast,  there  was  no  overall 
significant  difference  in  general  attitudes  towards  instructional  format  between  the  non-integrated  group  (M  = 
1.64)  and  integrated  group  (M  = 1.74),  with  both  groups  reporting  relatively  high  satisfaction  with  their 
respective  instructional  materials. 

The  third  research  question  sought  to  determine  if  the  management  of  redundancy  and  split  attention 
will  affect  psychomotor  performance.  Relative  to  psychomotor  performance,  the  integrated  group  scored 
significantly  higher  (M  = 39.76)  than  the  non-integrated  group  (M  = 30.85)  on  the  psychomotor  task 
performance,  F(l,39)  = 29.15,  p < .001,  MSe  = 27.90,  Es"=  .43. 

The  fourth  research  question  asked  what  aspects  of  the  psychomotor  skills  were  transferred 
immediately  following  instruction.  Overall,  the  integrated  group  scored  significantly  higher  (M  = 39.76)  than 
the  non-integrated  group  (M  = 30.85)  on  the  psychomotor  task  performance,  F(  1 ,39)  = 29.15,  p < .001,  MSe  = 
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27.90,  ES  = .43.  Specifically,  psychomotor  skills  related  to  technique  evaluation  (M  = 7.10):  F(  1,39)  = 20.23,  p 
< .001,  MSe  = 8.91,  ES  = .34;  and  the  application  section  (M  = 28.67):  F(l,39)  = 13.95,  p < .001,  MSe  = 13.37, 
ES  = .26  were  transferred  with  greater  efficiency,  without  practice  and  without  feedback  in  the  integrated 
instructional  group,  as  compared  to  control  group  participants  who  received  the  non  -integrated  instruction 
format  (M  = 2.90  and  24.40  respectively).  No  significant  differences  were  noted  on  the  comprehension  section 
between  groups  in  this  study. 


Discussion  and  Conclusions 

In  this  study,  there  was  an  overall  significant  difference  (Omnibus  F)  between  the  integrated  and  non- 
integrated  treatments  and  an  overall  moderate  effect  size  of  .52  was  observed.  Significant  results  were  found  in 
the  expected  direction  for  the  hypotheses  predicting  higher  achievement  on  the  written  examination  scores, 
lower  ratings  of  cognitive  load  following  instruction  and  technique  phases,  and  higher  performance  on  the 
psychomotor  task  by  the  integrated  treatment  group,  as  compared  to  the  non-integrated  treatment  group.  There 
were  no  significant  differences  noted  in  terms  of  the  time  required  to  complete  instruction  or  written 
examinations  between  groups. 


Hypothesis  one 

Hypothesis  one  predicted  that  participants  who  received  the  integrated  instructional  format  would 
achieve  higher  posttest  scores  as  compared  to  control  group  participants.  The  primary  variables  under 
assessment  were  written  posttest  scores,  which  entailed  the  further  analysis  content  structure  scores.  The  results 
for  cumulative  scores  and  content  structure  scores  indicated  that  there  was  a significant  difference  between  the 
two  instructional  conditions  in  the  expected  direction  with  the  integrated  instruction  group  scoring  significantly 
higher.  These  findings  suggest  that  the  instructional  complexity,  interactivity  of  elements,  and  novelty  of  the 
content  were  capable  of  placing  an  appreciable  load  on  the  learner’s  available  cognitive  resources.  These 
results  further  suggest  that  the  integrated  treatment  allowed  for  GCL  by  reducing  ECL  as  a function  of  sound 
design  practices.  Conversely,  these  results  also  suggest  that  the  traditional  treatment  (non-integrated  format) 
sufficiently  increased  ECL  and  sufficiently  limited  GCL  which  prevented  participants  from  developing  the 
appropriate  schema  and  understanding  of  the  content.  The  reduction  of  total  cognitive  load  via  the  management 
of  ECL  is  perhaps  the  most  prominent  cognitive  load  management  principle  and  is  consistent  with  findings 
identified  in  prior  research  (Bobis  et  ah,  1993;  Chandler  & Sweller,  1991,  1992;  Marcus,  Cooper,  & Sweller, 
1996;  Purnell  et  ah,  1991;  Tarmizi  & Sweller,  1988). 

Content  structure  scores 

Additional  analysis  was  conducted  to  assess  content  structure  scores  between  groups.  In  conditions 
where  the  instruction  required  mental  integration  (split  attention  effect)  for  understanding  or  in  situations  where 
instructional  materials  were  presented  with  redundant  features  (redundancy  effect)  the  integrated-instructional 
group  scored  significantly  higher  as  compared  to  the  control  group  on  respective  test  questions.  Because 
complex  learning  situations  composed  of  several  highly  interrelated  elements  will  create  the  heaviest  load  on 
working  memory,  the  differences  in  content  structure  scores  between  groups  provides  further  support  for  the 
preliminary  findings.  That  is,  the  content  containing  redundant  or  split  attention  features  represented  a 
discernable  difference  between  groups  in  terms  of  the  number  of  discrete  elements  that  participant’s  were 
required  to  maintain  and  manipulate  simultaneously  in  working  memory.  Furthermore,  the  lower  performance 
demonstrated  by  the  non-integrated  instruction  group  suggests  that  the  number  of  elements  exceeded  the 
processing  abilities  of  working  memory  and  sufficiently  limited  germane  cognitive  load.  Conversely,  the  higher 
performance  demonstrated  by  the  integrated  instruction  group  suggests  that  the  number  of  elements  did  not 
exceed  the  processing  abilities  of  working  memory  as  a function  of  sound  instructional  and  message  design 
practices.  Finally,  as  would  be  expected,  in  situations  where  it  was  not  necessary  for  the  learner  to  integrate 
divergent  sources  of  information  or  process  redundant  information,  there  was  no  difference  between  the 
integrated  and  non-integrated  formats  (Sweller  & Chandler,  1994;  Sweller  et  ah,  1998). 

Additional  Considerations 

As  a function  of  study  design,  some  meta-cognitive  processes  were  under  the  direct  control  of  the 
participants  and  subject  to  individual  strategies  that  were  in  part  a function  of  intrinsic  cognitive  load.  To  this 
end,  learners  may  or  may  not  have  utilized  advantageous  metacognitive  strategies.  Furthermore,  because  the 
diagrams  contained  in  both  treatments  reflected  procedural  sequences  as  a function  of  the  clinical  nature  of 


713 


localization  testing,  both  instructional  formats  would  have  offered  the  learner  an  isolated  step  by  step 
illustration  of  task  performance  and  an  overall  illustration  of  task  performance.  However,  while  each  isolated 
step  and  the  overall  strategy  driven  process  could  have  been  derived  from  either  instructional  format,  the 
redundant  and  divergent  sources  of  information  contained  in  the  non-integrated  group  combined  with  increased 
extraneous  load  decreased  GCL  would  have  made  it  difficult  for  the  non-integrated  treatment  group  to  form 
such  understandings.  Thus,  it  is  plausible  that  participant  access  and  execution  of  more  and  less  productive 
metacognitive  strategies  would  have  varied  between  groups  as  a function  of  instructional  format  and  available 
cognitive  resources. 

In  the  context  of  procedural  nature  of  the  treatment  materials  and  in  consideration  of  more  recent 
contributions  to  cognitive  load  theory,  the  integrated  group  may  have  chosen  to  learn  or  memorize  the 
individual  steps  in  isolation  (isolated  elements  approach  or  serial  processing)  before  attempting  to  integrate  the 
entire  process  (Pollock  et  al.,  2002).  While  the  claim  that  the  integrated  treatment  group  utilized  such  strategies 
is  speculative,  future  studies  might  query  the  participants  to  determine  what  type  of  metacognitive  strategies 
they  used  for  the  different  tasks  and/or  choose  a multi-stage  approach  in  order  to  manipulate  intrinsic  load. 

Hypothesis  Two 

Hypothesis  two  predicted  that  participants  who  received  the  integrated  instructional  format  would 
report  lower  subjective  ratings  of  cognitive  load  as  compared  to  those  receiving  the  non-integrated  format.  The 
primary  variables  under  assessment  were  subjective  ratings  of  cognitive  load  reported  after  completing  both  the 
instructional  unit  and  psychomotor  assessment.  The  integrated-treatment  group  reported  significantly  lower 
subjective  ratings  of  cognitive  load  post-instruction  and  post-psychomotor  assessment  as  predicted. 

These  significantly  lower  subjective  ratings  were  well  aligned  with  the  significantly  higher  objective 
performance  measures  achieved  by  the  integrated  treatment  group.  Additionally,  while  the  use  of  subjective 
ratings  of  cognitive  load  were  not  identified  in  prior  research  in  the  context  of  psychomotor  assessment  or  the 
performance  of  manual  physical  therapy  skills,  the  present  findings  suggest  that  such  measures  can  be  extended 
to  the  performance  of  psychomotor  tasks.  Specifically,  significantly  lower  subjective  ratings  of  cognitive  load 
reported  by  the  integrated  treatment  were  correlated  with  significantly  higher  psychomotor  assessment  scores  as 
discussed  below. 


Hypothesis  Three 

Hypothesis  three  predicted  that  participants  who  received  the  integrated  instructional  formats  would 
achieve  higher  performance  scores  on  manual  physical  therapy  tasks.  The  primary  variable  under  assessment 
was  cumulative  rubric  score  for  performance  of  manual  physical  therapy  skills.  On  this  task,  the  integrated- 
treatment  group  scored  significantly  higher  on  total  psychomotor  performance  in  the  expected  direction. 

As  a function  of  the  clinical  nature  of  localization  testing  and  its  practical  applicability  to  clinical 
practice,  the  psychomotor  rubric  consisted  of  three  distinct  sections:  (a)  comprehension,  (b)  application  and  (c) 
evaluation.  When  comparing  psychomotor  scores  between  groups,  the  integrated  group  scored  significantly 
higher  on  the  evaluation  section  and  the  application  section  of  the  rubric,  with  no  significant  differences  noted 
on  the  comprehension  section.  These  findings  suggest  that  both  groups  understood  the  concepts  presented  in 
their  respective  instructional  treatments,  though  only  the  integrated  treatment  group  was  able  to  demonstrate 
proficiency  on  psychomotor  tasks  performance.  These  specific  findings  could  be  attributed  to  both  the  content 
structure  of  the  two  treatments  and  the  level  of  complexity  of  the  content  itself.  Specifically,  the  presentation  of 
evaluation  and  application  content  structure  was  very  conducive  to  diagrammatic  presentation  and  in  fact,  the 
patient  positioning  phases  (application  and  evaluation)  and  the  application  phases  (procedural  steps)  were  both 
presented  in  diagrammatic  formats.  In  the  non-integrated  treatment,  the  participants  needed  to  integrate  the 
information  to  understand  the  procedure,  a constraint  that  was  not  present  in  the  integrated  treatment.  Plausible 
explanations  for  superior  performance  demonstrated  by  the  integrated  treatment  group  on  the  application  and 
evaluation  sections  were  specifically  discussed  above. 

In  terms  of  superior  performance  demonstrated  by  the  integrated  treatment  group  on  psychomotor 
tasks,  Romiszowski  (1993)  identified  that  psychomotor  learning  typically  involves  the  acquisition  of  both  skills 
and  knowledge  in  which  he  identifies  knowledge  as  “information  stored  in  the  performer’s  mind  or  available  to 
the  performer  in  some  reference  source”  and  skill  as  “actions  (intellectual  as  well  as  physical)  which  the 
performer  executes  in  a competent  manner  in  order  to  achieve  a goal”  (p.  130-131).  Romiszowski  additionally 
discusses  psychomotor  performances  across  a continuum  of  types  of  knowledge  content  in  which  he  notes  the 
distinctions  between  “reproductive  skills”  that  entail  repetitive  and  automated  actions  and  “productive  skills” 


714 


that  entail  the  use  of  adaptive  strategies  and  reasoning  skills.  This  study  employed  psychomotor  tasks  that  are 
consistent  with  Romiszowski’s  definition  of  “productive  skills”  as  the  participants  had  little  time  to  address 
repetition  or  automation  and  were  required  to  problem  solve  and  adapt  strategies  or  make  clinical  decisions 
during  the  psychomotor  assessment  phase  (Romiszowski,  1993). 

This  particular  type  of  skill  has  also  been  studied  by  Anderson  (1983)  in  his  development  of  the 
Adaptive  Control  of  Thought  (ACT)  model,  which  has  been  directed  towards  understanding  procedural 
knowledge  linked  to  cognitive  skills  relevant  to  decision  making  and  problem  solving  or  productive  skills. 

ACT  describes  that  “productions  provide  the  connection  between  declarative  knowledge  and 
behavior”(Anderson,  1983).  Relative  to  this  study,  Anderson’s  explanation  of  the  link  between  declarative 
knowledge  and  behavior  helps  to  offer  further  explanation  for  the  significantly  superior  performance  on  task 
performance  by  the  integrated  instruction  group  as  compared  to  the  non -integrated  instruction  group. 
Specifically,  given  the  significantly  superior  performance  demonstrated  by  the  integrated  instruction  group,  the 
non-integrated  instruction  group  would  have  likely  experienced  a waterfall  failure  effect  in  that  they  were 
unsuccessful  in  transferring  the  necessary  declarative  knowledge  and  problem  solving  schema  to  long  term 
memory  that  were  required  for  successful  task  performance. 

Hypothesis  Four 

Hypothesis  four  predicted  that  participants  who  received  the  integrated  instructional  formats  would 
have  lower  task  completion  times.  The  primary  variables  under  assessment  were  task  completion  times  for  the 
instructional  unit  and  the  written  examination.  There  were  no  significant  differences  between  the  groups  on 
time  spent  to  complete  the  instructional  unit  or  the  written  exa  mination.  These  two  variables  had  observed 
power  values  of  .055  and  .408,  respectively,  which  means  that  many  more  participants  would  have  been  needed 
to  detect  a significant  difference,  if  it  in  fact  existed.  In  contrast,  Stem-and-Leaf  Plots  for  the  variable  “Total 
Exam  Time”  in  both  the  non-integrated  and  integrated  instructional  groups  presented  two  data  plots  with 
extreme  outliers  relative  to  all  other  data,  which  contained  no  extreme  outliers.  Specifically:  non-integrated 
format  (1.00  extreme.  >/=  26.0)  and  integrated  format  (1.00  extreme,  >/=  30).  When  additional  analysis  was 
performed  with  these  two  extreme  outliers  removed.  Total  Exam  Time  was  reported  as  F(l,  37)  = 5.11  , p = .03, 
ES  = . 12.  One  plausible  explanation  for  these  finding  could  be  attributed  to  sample  size,  as  a much  larger 
sample  size  may  have  not  been  so  readily  influenced  by  extreme  outliers,  if  present.  Additionally,  another 
plausible  explanation  is  that  the  integrated  instruction  group  had  to  give  little  mental  effort,  while  the  non- 
integrated  group  felt  overwhelmed  and  did  not  give  the  additional  effort  needed  to  overcome  the  limitations  of 
the  materials  needed  to  promote  learning  with  understanding.  Future  studies  might  query  participants  to 
determine  affective  or  motivational  responses  to  varied  instructional  formats  in  order  to  better  understand 
underlying  cognitive  reasoning. 


Research  Questions 

The  first  research  question  asked  if  redundancy  and  split  attention  principles  derived  from  Cognitive 
Load  Theory  were  transferable  to  the  knowledge  domain  of  manual  physical  therapy.  In  this  study,  the 
statistically  significant  results  in  the  expected  direction  for  three  of  four  hypotheses  provide  support  for  the 
transferability  of  cognitive  load  theory  design  principles  to  the  teaching  of  manual  physical  therapy. 

The  second  research  question  asked  if  instructional  materials  designed  in  accordance  with  cognitive 
load  theory  design  principles  would  positively  influence  learner  attitudes  towards  instruction.  In  this  study, 
statistically  significant  results  in  the  expected  direction  indicated  that  attitudes  as  a function  of  subjective 
ratings  of  cognitive  load  reported  by  the  integrated  instruction  group  were  positively  influenced  as  compared  to 
the  non-integrated  instruction  group.  However,  general  attitudes  towards  instructional  formats  did  not  identify 
an  overall  significant  difference  between  the  integrated  and  non-integrated  groups.  In  this  context,  both  groups 
perceived  both  units  of  instruction  rather  favorably  which  for  the  non-integrated  treatment  group  was  in  contrast 
to  both  objective  measures  and  subjective  ratings  of  cognitive  load.  In  this  study,  the  short  instructional  time 
might  have  also  influenced  the  subjective  ratings  while  longer  instructional  periods  noted  in  a traditional 
classroom  setting  might  have  provided  different  findings. 

The  third  research  question  asked  if  the  management  of  redundancy  and  split  attention  would  affect 
psychomotor  performance.  Statistically  significant  results  in  the  expected  direction  indicated  that  cognitive 
load  theory  design  principles  were  successfully  extended  to  the  performance  of  manual  physical  therapy  skills. 
Additionally,  no  differences  were  found  in  conditions  involving  low  element  complexity  and  interactivity,  such 
as  with  conditions  that  would  not  be  expected  to  tax  working  memory  resources. 
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The  fourth  research  question  asked  what  aspects  of  the  psychomotor  skills  are  transferred  immediately 
following  instruction;  accurately  without  practice  and  without  feedback.  Study  findings  suggest  that  superior 
performance  on  both  cognitive  and  psychomotor  performance  can  be  obtained  as  a function  of  instructional 
message  design  principles  that  eliminate  split  attention  and  redundancy  effects. 

Applications  to  Physical  Therapy 

Relative  to  practical  and  clinical  significance  is  the  observation  that  the  integrated  treatment  group 
achieved  an  89%  on  the  examination  and  90%  on  the  psychomotor  assessment  (practical  examination),  while 
the  non-integrated  treatment  group  achieved  a 79%  and  70%  respectively,  the  latter  grades  would  be  considered 
failing  by  program  standards.  Additionally,  practical  examinations  (formal  psychomotor  assessments)  are  often 
limited  to  a single  “re -take”  opportunity  prior  to  course  and/or  program  dismissal.  To  this  end,  the  differences 
in  scores  from  a curricular  perspective  as  a function  of  instructional  format,  as  well  as  the  direct  applicability  of 
the  treatment  materials  to  real  world  clinical  practice  are  salient  features  of  this  study. 

Relative  to  cognitive  load  theory,  distinctions  between  the  reduction  of  ECL  and  the  “freeing  up”  of 
GCL  via  message  design  strategies  and  the  external  facilitation  of  GCL  via  schema  driven  strategies  could  be 
better  delineated  via  multiple  experiment  designs.  Second,  the  applicability  of  isolated  and  interacting  elements 
approaches  to  cognitive  and  psychomotor  tasks  should  be  entertained.  Last,  future  research  should  attempt  to 
determine  to  what  extent  (if  any)  metacognitive  strategy  selection  is  influenced  by  manipulating  various  facets 
of  cognitive  load  and  the  effects  that  isolated  and  interacting  elements  approaches  for  both  cognitive  and 
psychomotor  tasks. 


Conclusions 

This  study  used  ecologically  valid  materials  in  a realistic  classroom  setting.  As  predicted,  the 
participants  receiving  the  integrated-instructional  materials  scored  significantly  higher  on  the  written  posttest. 
This  result  suggests  that  designers  can  increase  the  germane  cognitive  load  by  reducing  the  extraneous  cognitive 
load  through  good  instructional  and  message  design  practices.  This  study  extended  the  prior  research  by 
examining  the  effect  of  lowered  extraneous  cognitive  load  on  the  performance  of  a psychomotor  task.  The 
significant  increase  in  performance  by  the  integrated-materials  treatment  participants  suggests  that  psychomotor 
performance  is  also  enhanced  by  an  increase  in  germane  cognitive  load  capacity. 

These  findings  suggest  that  instructional  designers  should  reduce  designs  that  create  split-attention  and 
redundancy.  In  addition,  graphics  should  be  created  using  an  integrated  approach  suggested  by  Sweller  (1999) 
that  further  reduces  extraneous  cognitive  load.  With  complex  information,  it  is  apparent  that  instructional 
designers  must  manage  extraneous  cognitive  load  to  afford  learners  the  opportunity  to  develop  appropriate 
schemata.  The  findings  from  this  study  combined  with  the  robust  nature  of  cognitive  load  theory  in  general, 
warrants  further  investigation  of  cognitive  load  principles  in  the  design  of  Physical  Therapy  instructional 
materials  and  the  application  of  cognitive  load  principles  to  psychomotor  performance. 
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Abstract 

This  paper  reports  the  findings  of  an  exploratory  study  that  investigates  what  Technology 
Coordinators  (TC)  are  expected  to  do  and  what  they  actually  do.  Schools  have  invested  large  sums  of  money  on 
technology  and  they  have  high  expectations  for  the  educational  outcomes  associated  with  the  use  of  such 
technology.  A c/uestion  then  arises  regarding  the  expectations  of  the  TC  whose  job  it  is  to  coordinate  or 
manage  the  school’s  technology.  What  are  the  expectations  for  the  TC?  Do  the  expectations  match  what  the 
TC  actually  does?  Our  exploratory  study  examines  the  expectations  and  realities  in  detail.  The  results  of  this 
exploratory  study  indicate  that  there  is  a disconnect  between  the  expectations  and  the  reality  of  the  TC’s  job. 

We  present  three  main  findings.  First  for  a TC  to  succeed,  they  must  have  a clear,  yet  dynamic,  job  description 
that  is  widely  disseminated  within  the  educational  community.  Second,  job  descriptions  should  vary  depending 
on  the  needs  of  the  TC’s  educational  community.  Third,  the  TC  needs  time  built  into  their  schedule  to  afford 
them  the  opportunity  to  work  on  professional  development  (PD)  projects. 

Background 

According  to  the  National  Center  for  Educational  Statistics,  the  average  public  school  has  124 
instructional  computers  (NCES,  2003),  and  99%  of  all  public  schools  have  access  to  the  Internet  (NCES,  2002). 
The  teacher  with  an  interest  in  computers  is  ordinarily  responsible  for  integrating  the  instructional  computers 
into  the  curriculum.  These  teachers  typically  find  themselves  playing  the  role  of  the  school  or  district  TC. 1 We 
examine  the  roles  of  TCs  to  determine  if  the  expectations  match  the  reality  of  what  the  TCs  are  able  to 
accomplish. 

The  literature  on  the  TC’s  job  responsibilities  exemplifies  the  diverse  views  of  the  TC’s 
responsibilities.  The  main  area  of  agreement,  across  the  literature,  as  to  the  TC’s  role  is  that  of  PD  and 
technician  (Strudler,  1999;  Marcovitz,  1998:  Reilly,  1999;  Lai,  Trevern  & Pratt,  2002).  The  other  combined 
roles  of  the  TC  include:  Classroom  Assistant,  Curriculum  Consultant,  Curriculum  Designer,  Policy  Maker, 
Strategic  Planning,  Manager  and  Envisioner  (Strudler,  1999;  Marcovitz,  1998:  Reilly,  1999;  Lai,  Trevern  & 
Pratt,  2002).  This  suggests  that  the  expectations  for  TCs  vary.  The  question  is  then:  why  so  much  variance? 

Before  we  address  the  question  of  the  variance  in  job  responsibilities,  it  is  important  that  we  discuss 
effective  PD  for  technology  utilization  in  educational  environments.  PD  was  one  area  that  the  literature  agreed 
was  one  of  the  TC’s  job  responsibilities.  We  believe  that  PD  is  an  evolutionary  process  and  should  be  based  on 
the  technology  needs  of  the  community.  Over  time  the  method  for  professional  development  will  change  as  the 
technological  skills  and  needs  of  the  educational  community  change.  In  other  words,  technology-related  PD  is  a 
process  that  changes  as  teachers  and  students  become  more  versed  in  technology  (Sherry,  Billig,  Tavaline,  & 
Gibson,  2000).  There  may  be  so  much  variance  in  the  literature  regarding  the  role  of  the  TC  because  the  needs 
for  PD  change  over  time. 

This  study  seeks  to  determine  what  Technology  Coordinators  do  and  what  others  in  their  environment 
expect  of  them.  We  seek  to  determine  if  there  is  a disconnect  between  expectations  and  reality  within 
educational  environments  and  across  environments.  We  believe  there  are  two  possible  reasons  for  a disconnect. 
The  first  explanation  is  that  the  technology  needs  of  the  educational  communities  are  diverse  which  accounts  for 
the  diversity  in  roles  documented  in  the  literature.  If  the  first  explanation  is  correct  then  the  expectations  within 
an  individual  TC’s  environment  should  match  the  reality  of  the  job.  The  second  explanation,  for  the  wide 
variance  in  the  roles  of  TCs  reported  in  the  literature,  is  that  there  are  simply  too  many  expectations  for 
technology  coordinators  due  to  the  varied  ways  in  which  technology  is  employed  in  the  educational  setting.  If 
this  second  explanation  is  true,  then  expectations  will  not  match  the  reality  of  the  TC’s  jobs  within  and  across 
educational  communities. 

We  believe  that  everyone  within  a given  educational  community  needs  to  have  a clear  understanding 
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of  what  the  TC  is  expected  to  do  to  avoid  inefficient  and  ineffective  use  of  the  technology.  If  there  is  a 
disconnect,  then  educational  communities  will  need  to  determine  ways  to  alleviate  the  disconnect  and  work 
towards  a unified  vision  of  reasonable  expectations.  A unified  vision  will  allow  schools  to  design  the  TC’s  role 
so  that  the  TCs  will  fulfill  the  educational  community’s  needs  while  avoiding  inefficient  and  ineffective  use  of 
technology. 


It  is  important  to  note  that  we  use  the  title  “Technology  Coordinators”  to  refer  to  the  individual  who  is 
responsible  for  coordinating  and/or  managing  technology  efforts  within  an  educational  environment. 

Depending  on  the  school  system,  this  person  can  be  school-based  or  system-based.  Some  common  titles  for 
such  positions  are  Computer  Coordinator,  Technology  Facilitator,  and  Technology  Coordinator.  We  refer  to 
these  positions  in  general  and  we  realize  that  there  are  fundamental  differences  in  school-based,  district-based 
and  system-based  positions.  This  is  an  exploratory  investigation  that  is  looking  at  general  expectations  for  those 
who  coordinate  technology  efforts  within  educational  environments. 


Methodology 

A descriptive,  exploratory  case  study  was  conducted.  To  triangulate  the  results,  a variety  of  data 
collection  techniques  were  used  and  information  was  gathered  from  a variety  of  sources.  What  follows  is  a 
discussion  of  the  participants  and  the  collection  methods. 

Participants 

Primary  Participants  Two  district  TCs  and  one  middle  school  TC  participated.  TCs  were  selected 
based  upon  their  willingness  to  participate  and  their  proximity  to  the  primary  investigator’s  office. 

Secondary  Participants  In  addition  to  the  TCs,  four  people  (including  a supervising  administrator  and 
a media  coordinator  who  is  at  the  same  level  as  the  TC)  from  each  TC’s  school/school  district  participated  in 
interviews. 

The  secondary  participants  are  as  follows: 

Middle  School  TC:  the  Middle  School  Principal,  the  Middle  School  Media  Coordinator  and  two 
teachers  from  the  middle  school. 

District  TC  #7:  the  High  School  Principal,  the  District  Wide  Media  Coordinator,  the  High  School  TC 
and  a High  School  Teacher. 

District  TC  #2:  the  Assistant  Superintendent  of  the  District,  the  District  Wide  Media  Coordinator  and 
two  elementary  school  media  coordinators. 

By  interviewing  people  who  play  different  roles  in  relation  to  the  TC  we  are  able  to  get  a greater 
perspective  to  enhance  our  understanding  of  the  TC’s  activities. 

Interviews 

Primary  Participants  Prior  to  the  interview,  each  TC  received  a copy  of  the  interview  questions.  The 
goal  of  the  interview  was  to  get  acquainted  with  the  TC  and  gain  an  understanding  of  their  role.  The  initial 
interview  was  approximately  one-hour  in  length.  Several  days  after  the  initial  interview  a half-day  observation 
and  a follownip  interview  were  conducted.  The  purpose  was  to  clarify  any  ambiguities  from  the  initial 
interview  and  to  allow  each  TC  an  opportunity  to  expand  upon  their  initial  responses.  A second,  half-day 
observation  was  scheduled  after  the  follow-up  interview. 

Subsequent  to  the  second  half-day  observation  a summary  of  the  interview  notes  were  sent  to 
each  participant  for  validation. 

Secondary  Participants  Interviews  took  place  after  the  initial  interview  with  the  TC.  Each  secondary 
participant  received  the  interview  questions  prior  to  the  interview.  The  purpose  of  these  half-hour  interviews 
was  to  learn  how  the  TC  interacts  with  their  colleagues  and  their  perceptions  of  the  TC’s  role  in  the  educational 
environment.  The  teachers  and  the  media  coordinators/specialists  were  asked  the  same  questions  while  the 
administrators  were  asked  two  additional  questions  regarding  the  TC’s  responsibilities. 

The  information  gathered  from  the  interviews  was  recorded,  synthesized,  interpreted  and  sent  to  the 
corresponding  study  participant  for  validation  purposes. 

Observations 

After  the  initial  interview,  each  TC  was  shadowed  for  two  half-day  sessions.  After  each  half-day 
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shadowing,  observation  notes  were  synthesized  and  sent  to  the  TC  for  comment. 


Documents 

During  the  initial  interview,  each  TC  was  asked  for  a copy  of  their  official  job  description  and  any 
other  material  that  would  provide  insight  into  their  responsibilities  (such  as  promotional  materials  for  the 
school).  Only  the  two  district  TCs  had  official  job  descriptions.  Both  of  these  descriptions  listed  a variety  of 
activities  and  responsibilities.  The  brief  descriptors  were  combined  into  a list  of  20  descriptors  (see  table  A). 
Each  of  the  TCs  and  administrators  were  asked  to  determine  which  of  the  tasks  the  TC  is  responsible  for  and  to 
rank  the  top  five. 


Analysis 

The  investigator’s  write-ups  of  the  teacher  interviews  were  sent  to  each  respective  teacher  for 
comment.  All  but  two  teachers  returned  the  investigator’s  write-ups.  The  corrections  the  investigator  received 
were  confined  to  the  spelling  of  names  and  clarification  of  titles. 

Once  the  majority  of  the  respondents'  comments  on  the  write-ups  were  received,  data  from  each  of  the 
sources  and  data  gathering  methods  were  coded.  The  Constant  Comparison  Method  was  used  to  code  the  data. 
The  coded  data  was  then  analyzed  for  trends,  relationships,  and  linkages  between  the  codes.  The  trends  were 
then  synthesized  and  an  expert  in  the  field  commented  on  the  synthesis.  A copy  of  the  final  write-up  was  sent 
for  comment  to  each  TC. 


Results 

Document  Analysis 

Table  A lists  the  combined  job  descriptors  for  the  two  job  descriptions  provided  by  the  participants. 
The  columns  next  to  the  descriptors  are  the  respondents’  rankings  for  each  descriptor  with  the  TC  listed  first 
and  the  TC’s  supervisor  immediately  following.  The  numerical  rankings  are  as  follows.  The  most  important 
descriptor  is  assigned  a value  of  1,  the  second  a 2,  and  so  on  up  to  5.  An  X indicates  that  the  job  descriptor  is 
considered  a part  of  the  TC’s  job  but  was  not  considered  one  of  the  top  5 descriptors.  A blank  indicates  that  the 
respondent  did  not  consider  the  job  descriptor  as  part  of  the  TC’s  job. 


Table  A:  Perceived  Job  Descriptors  Document 

School 

District 

Descriptor 

□ 

la 

B 

2a 

m 

3a 

Supervises  and  maintains  operation  of  voice,  data,  video,  and  other  technology  systems. 

m 

i 

m 

X 

□ 

X 

Advocates  technology  utilization,  awareness,  and  assists  building  technology  committees. 

X 

2 

m 

5 

Services  and  maintains  related  technologies  and  the  wired  infrastructure  of  the  schools. 

X 

X 

Maintains  the  school’s  on-line  bulletin  board  services. 

X 

X 

Formulates  and  monitors,  with  the  technology  committee,  a vision  and  a plan  for  technology 
implementation. 

X 

3 

X 

1 

1 

X 

Coordinates  distance  learning  activities. 

X 

Coordinates,  evaluates,  and  recommends  software  purchases  and  replacement. 

H 

5 

m 

X 

X 

Maintains  an  inventory  of  technology  for  school  computers. 

B 

X 

m 

X 

H 

Plans  and  supervises  technology  maintenance. 

X 

B 

3 

Plans  and  provides  staff  development  and  training  opportunities 

a.)  Assists  teachers  with  hardware  and  software  problems,  b.)  Trouble  shooter  and  repair 
technician  when  appropriate,  c.)  Assist  the  teachers  in  optimizing  the  integration  of 
technology  and  instruction. 

3 

2 

1 

5 

1 

Develops  and  monitors  specifications  for  technology  purchases. 

B 

X 

a 

3 

m 

X 

Trouble  shooter  and  repair  technician  when  appropriate. 

i 

X 

X 

Management  of  vendor  relationships,  including  maintenance,  hardware  and  software 
upgrades  and  ongoing  development  work  with  the  system  users. 

X 

X 

i 

5 

X 

X 

Other  duties  as  assigned  by  the  superintendent  or  his  designee. 

B 

X 

H 

B 

Plan  offerings  for  xxx  schools  adult  education  concerning  technology. 

n 
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Assist  in  development  of  building  office  automation  plans. 

H 

a 

Disseminate  info,  about  new  tech.,  software,  meetings,  and  classroom  apps.  of  comps,  to 
appropriate  personnel. 

■ 

X 

X 

2 

X 

Realize  latest  technologies  and  their  educational  applications  by  attending  conferences, 
workshops  and  meetings. 

X 

X 

3 

X 

X 

2 

Participate  in  state  and  national  professional  associations  and  encourage  participation  of  other 
district  personnel. 

X 

X 

X 

X 

X 

■ 

Disseminate  info,  about  technology  usage  in  xxx  schools  to  parents,  community  members, 
students  and  faculty. 

X 

■ 

X 

X 

X 

Agreement  within  school  or  district 

83% 

58% 

73% 

Agreement  across  district 

50% 

Interview  Results 

The  following  are  the  summaries  of  all  of  the  interviews  that  were  conducted. 

Professional  Development  and  Communication  Skills  The  TC  needs  to  freely  provide 
information/instruction  to  others  and  not  withhold  information.  The  comfort  level  of  the  community  improves 
as  they  receive  more  information  and  instruction.  As  one  participant  stated  “the  TC  needs  to  be  able  to  share 
(their)  knowledge  and  empower  others  to  become  competent  technology  users.”  Additionally,  the  TC  needs  to 
communicate  and  interact  with  people  on  a variety  of  levels  and  they  need  to  be  approachable  and  friendly  so 
that  everyone  is  be  comfortable  seeking  their  assistance  and  training. 

Well  Versed  in  Technology  and  Education  The  majority  of  the  interviewees  stated  that  the  TC  needed 
to  be  well  versed  in  how  the  school’s  technological  systems  operate,  as  well  as  how  people  within  the  systems 
operate.  With  a solid  understanding  of  these  systems,  the  TC  will  be  able  to  succeed  in  maintaining  and 
operating  the  technology. 

One  administrator  noted  that  “while  school  corporations  might  want  to  move  the  Technology 
Coordinator’s  responsibilities  away  from  working  on  technical  aspects  and  more  towards  curriculum 
integration,  it  is  a difficult  task  because  many  Technology  Coordinator’s  salaries  come  out  of  the  capital  project 
funds,  which  requires  that  they  work  on  maintaining  the  school  corporation’s  hardware/software.” 

Another  person  noted,  “while  the  district  wide  Technology  Coordinator  excels  in  integrating  technology  into  the 
curriculum,  she  still  spends  a large  portion  of  her  time  working  on  the  upkeep  and  trouble-shooting  of  the 
technology.”  This  is  mainly  due  to  the  way  the  technology  support  system  is  set  up.  It  forces  people  in  the 
school  system  to  call  upon  the  district-wide  TC  for  technical  maintenance. 

Stressful  job  Many  of  the  interviewees  indicated  that  the  TC’ s job  could  be  very  stressful  due  to  the 
wide  variety  of  technologies  and  wide  range  of  people  with  whom  they  interact.  They  need  to  be  flexible  in 
order  to  “switch  gears”  due  to  the  nature  of  the  job  and  the  wide-variety  of  responsibilities.  For  instance,  one 
person  stated  “the  TC  is  the  person  that  the  teachers  and  staff  go  to  for  assistance  with  technology,  ranging  from 
resources  to  equipment  requests,  or  training.” 

To  deal  with  the  stress  of  the  job  some  interviewees  emphasized  the  ability  of  the  TC  to  prioritize  their 
responsibilities  so  that  the  most  immediate  needs  are  served  first.  One  interviewee  stated  “they  are  being  asked 
a question  on  a different  topic  every  ten  minutes.”  Another  stated  the  TC  “needs  to  be  on  call  to  assist  teachers 
when  there  are  problems.”  Many  of  the  interviewees  stated  that  it  is  important  for  the  TC  to  refer  questions 
and/or  requests  to  others.  Not  all  of  the  technology  problems  need  to  be  solved  by  the  TC  personally.  The  belief 
is  that  if  the  TC  is  flexible  and  delegates  tasks  then  their  workload  will  become  manageable. 

Office  locale  Many  interviewees  stated  that  it  was  important  to  have  access  to  the  TC  at  all  times  to 
solve  problems  as  they  arise.  Some  felt  that  a key  element  to  properly  integrating  the  technology  into  the 
classroom  is  to  have  the  person  responsible  for  the  integration  of  the  technology  become  an  active  participant  in 
that  environment.  Thus,  it  is  important  to  locate  the  coordinator’s  office  in  the  same  building  if  possible. 
Additionally  having  the  TC  in  the  same  building  opens  the  lines  of  communication.  Having  the  TC's  office 
within  close  proximity  allows  the  school  to  capitalize  on  their  skills  to  a greater  degree  than  a school  that  does 
not  have  easy  access  to  the  coordinator. 
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Visionary  There  were  a few  interviewees  who  stated  that  the  district  wide  TC  is  a "visionary”  who 
plans  for  future  technologies  within  the  educational  environment.  These  plans  include  technology  investments 
as  well  as  plans  for  adapting  the  technology  to  keep  it  current. 

Observation  Results 

Observations  See  Appendix  A for  illustrative  transcripts  of  two  observations.  The  number  of 
transcripts  presented  in  the  appendix  has  been  limited  to  conserve  space.  Pseudonyms  are  used  to  maintain 
anonymity. 

Technology  The  TCs  are  responsible  for  a wide  range  of  technologies.  They  included  video  retrieval, 
voice  systems  and  computer  systems  (hardware  and  software,  instructional  and  administrative).  The  software 
included:  attendance,  communication,  productivity  and  instructional  software. 

Discussion 

The  following  summarizes  one  of  the  themes  that  grew  out  of  our  study. 

"We  weren't  trained  to  work  on  technology,  we  were  trained  to  educate.  We  need  to  get  back  to  that- 

spending  the  majority  of  our  time  to  work  with  education  rather  than  the  majority  of  the  time  working 

on  the  equipment."  District  Technology  Coordinator 

It  was  apparent  from  the  results  of  the  job  description  analysis  and  the  interviews  that  all  of  those 
involved  in  the  study  believed  that  the  primary  responsibilities  were  to  assist  teachers  with  integrating 
technology  into  the  curriculum  and  be  responsible  for  the  maintenance  of  the  school  environment’s  hardware 
and  software. 

Job  descriptions  As  discussed  in  the  results  section,  the  TCs  and  administrators  could  agree  on  only 
one  out  of  the  20  constructs  that  could  be  considered  to  be  one  of  the  TC’s  top  five  responsibilities.  This 
construct  is:  "Plans  and  provides  staff  development  and  training  opportunities  a)  assists  teachers  with  hardware 
and  software  problems,  b)  trouble  shooter  and  repair  technician  when  appropriate,  and  c)  assist  the  teachers  in 
optimizing  the  integration  of  technology  and  instruction."  This  description  incorporates  both  technology 
integration  and  responsibility  for  repairing  and  trouble-shooting  technical  problems. 

A second  construct  that  all  but  one  of  the  TCs  and  administrators  rated  among  the  top  five  priorities 
was,  "advocates  technology  utilization,  awareness,  and  assists  building  technology  committees." 

It  was  clear  from  the  job  descriptions  that  the  primary  responsibilities  of  the  TCs  within  and  across 
communities  are  ensuring  that  the  hardware  and  software  is  operational  and  PD.  This  was  also  consistently 
evident  during  the  interviews  and  observations. 

However,  when  it  came  to  agreement  within  the  educational  community  regarding  the  overall 
responsibilities,  there  did  not  appear  to  be  much  agreement  between  the  TC  and  the  supervisor  (83%,  58%,  and 
73%  agreement  respectively).  Furthermore  there  was  only  50%  agreement  across  the  two  school  districts.  This 
indicates  that  while  there  is  agreement  in  regards  to  the  primary  responsibilities,  the  expectations  for  secondary 
responsibilities  are  not  clear  within  or  across  educational  environments. 

Perceptions  Everyone  interviewed  stated  that  the  TC  needs  to  be  able  to  solve  technical  problems  but 
they  also  need  to  be  able  to  teach  others  how  to  use  the  technology  in  an  educational  setting.  PD  was  perceived 
to  be  a very  important  part  of  the  TC’s  job.  This  perception  is  consistent  with  the  findings  of  the  job 
descriptions.  However,  there  seemed  to  be  a disconnect  between  these  responsibilities  and  how  coordinators 
spent  their  time. 

During  the  interview  portion  of  the  study  only  one  TC  agreed  with  their  administrator  on  what  their  job 
description  was.  This  TC  didn’t  have  an  official  job  description.  The  unwritten  yet  commonly  understood 
description  was  that  the  TC  "...would  be  responsible  for  anything  that  has  a bell,  button  or  buzzer  on  it." 
However,  when  asked  to  rank  the  job  descriptors  this  TC  and  supervisor  achieved  83%  agreement.  The  other 
two  TCs  had  wide  variance  between  the  perceptions  of  the  administrator  and  the  TCs  with  regards  to  their 
responsibilities.  Without  a clear  definition  of  what  the  TC  is  expected  to  do,  there  is  a greater  likelihood  for 
coordinators  to  focus  on  only  the  most  glaring  problems  first  and  then  tend  to  other  responsibilities  when  time 
allows.  This  was  rather  apparent  during  the  observations. 

Another  cause  of  conflict  is  a question  of  the  funds  for  which  the  TC’s  salary  is  paid.  In  two  of  the 
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schools  the  TC’s  salary  was  paid  out  of  the  capital  projects  fund,  which  is  the  fund  for  the  maintenance  and 
upkeep  of  the  school  building,  furniture  and  equipment-including  technology.  In  these  instances  it  was 
necessary  that  the  TC  spend  the  majority  of  their  time  on  the  maintenance  and  upkeep  of  the  technology.  A 
problem  arises  when  the  TC  is  expected  to  assist  with  the  integration  of  the  technology  into  the  curriculum. 
Every  participant  in  the  study  indicated  that  integration  was  expected  of  the  TC. 

Wide  Variance  in  Clients  Abilities  and  Needs  Throughout  the  observations  and  the  interviews  it  was 
apparent  that  the  TC  had  to  work  with  a wide  range  of  people.  They  assisted  teachers  who  varied  with  regards 
to  their  technical  skills,  students,  administrators,  secretarial  staff,  parents  and  other  members  of  the  community. 

Problems  Associated  with  Being  "On  Call"  The  TCs  stated  that  they  needed  to  be  able  to  quickly  solve 
problems  when  a technical  "emergency"  arises.  This  idea  was  also  stressed  in  the  interviews  with  the  TC’s 
colleagues.  There  was  an  emphasis  on  the  idea  that  the  technology  needed  to  be  operational  in  order  for  it  to  be 
used— if  it  does  not  work  no  one  can  use  it.  As  one  interviewee  stated,  if  the  technology  is  unstable  the  staff  will 
not  use  it  because  it  will  cause  them  to  lose  their  "faith  in  the  technology.”  It  is  unlikely  that  anyone  would 
argue  against  the  idea  that  the  technology  needs  to  be  stable  and  operational.  The  concern  is  when  the  TC  has 
to  determine  when  maintenance  takes  priority  over  PD  or  another  curriculum  related  responsibility. 

Observations  indicated  that  the  TCs  day  was  fragmented.  People  were  constantly  stopping  by  the 
office  or  stopping  the  TC  in  the  hall  to  ask  questions.  Some  of  these  questions  did  not  need  to  be  answered 
immediately  but  because  the  TC  was  "accessible,"  individuals  asked  questions  regardless  of  what  the  TC  was 
engaged  in  at  the  time.  Additionally,  it  is  important  to  note  that  the  TCs  thrived  on  being  able  to  solve  problems 
as  they  arise.  Each  appeared  to  be  very  good  at  what  they  do  and  appeared  to  have  an  excellent  rapport  with 
those  with  which  they  worked.  These  were  important  characteristics  that  were  noted  in  the  interviews.  While 
the  TC’s  day  was  fragmented  due  to  issues  associated  with  being  ‘‘on  call,”  one  TC  was  observed  preparing  for 
a PD  activity  and  another  conducted  a training  session  for  members  of  the  Chamber  of  Commerce. 

In  addition,  each  of  the  TCs  developed  coping  mechanisms  for  their  fragmented  day,  such  as  requiring 
individuals  to  send  their  questions  or  requests  via  e-mail.  Or,  establishing  a ‘‘response  tree,”  where  a teacher 
with  advanced  technology  skills  is  the  first  point  of  contact,  then  the  TC.  While  these  techniques  are  effective 
to  a degree,  there  are  those  who  would  prefer  to  work  directly  with  the  TC.  The  interviews  made  it  clear  that 
the  expectation  was  that  the  TC  needed  to  be  ‘‘on  call”  in  order  to  answer  questions  and  solve  problems.  Also 
based  on  the  observations  there  were  enough  interruptions  in  the  TC’ s day  to  make  it  difficult  to  spend  a 
reasonable  amount  of  time  on  technology  integration  projects. 

Office  locale  The  physical  location  of  the  TC’s  office  seems  to  have  an  impact  on  how  many 
interruptions  or  requests  they  have  during  the  day.  One  TC’s  office  was  in  a large  room  in  the  back  corner  with 
their  assistants’  desk  directly  off  the  entrance  to  the  main  room.  The  assistant  served  as  a gatekeeper  for  the 
coordinator.  She  took  care  of  the  computer  lab  scheduling,  answered  some  of  the  walk-in  traffic  questions, 
answered  phone  requests  and  during  her  spare  time  she  evaluated  educational  software.  The  TC  was  still 
interrupted  however  the  interruptions  were  minimal  as  compared  to  the  other  two  TCs,  who  did  not  have 
gatekeepers.  The  other  two  TCs’  offices  were  in  high  traffic  areas  and  they  did  not  have  a person  who  served  as 
a filter  for  technology  questions.  Each  of  these  coordinators  indicated  that  their  offices  were  going  to  be  moved 
soon  to  a locale  that  has  lower  walk-in  traffic.  Each  was  looking  forward  to  their  move  because  they  felt  that  it 
would  afford  them  more  time  to  work  on  educational  activities. 

Responsibility  for  Technology  The  simple  fact  is  that  the  more  technology  that  the  TC  oversees  the 
greater  likelihood  that  they  will  spend  their  time  maintaining,  repairing  or  upgrading.  As  noted  in  the  results 
section,  each  of  the  TCs  had  a large  variety  of  software  and  equipment  for  which  they  were  responsible.  A 
large  quantity  and  variety  of  technology  can  influence  the  amount  of  time  that  the  TC  has  to  spend  on 
maintaining  the  technology  rather  than  on  curriculum  integration. 

Conclusion  and  Recommendations 

The  purpose  of  this  research  is  to  determine  if  there  is  a disconnect  between  what  TCs  are  expected  to 
do  and  what  they  actually  do.  This  is  one  inquiry  in  the  greater  pursuit  of  efficient  and  effective  utilization  of 
technology  in  educational  environments.  The  results  of  this  exploratory  case  study  indicate  that  the  primary 
responsibilities  for  TCs  are  to  integrate  technology  into  the  curriculum  and  to  be  responsible  for  the 
maintenance  and  upkeep  of  the  school/district’s  hardware  and  software.  There  wasn’t  much  agreement  amongst 
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the  participants  within  or  across  educational  communities,  regarding  other  responsibilities  and  the  expectations 
for  attaining  these  primary  responsibilities.  Additionally,  the  reality  of  the  job  appears  to  indicate  that  the 
majority  of  the  TC’s  time  is  spent  on  the  maintenance  and  upkeep  of  the  school/district’s  hardware  and 
software,  leaving  a small  portion  of  their  time  for  technology  integration  projects. 

Clear  job  descriptions  need  to  be  developed  to  match  the  needs  of  the  educational  environment.  These 
job  descriptions  need  to  be  dynamic  to  meet  the  ever-changing  needs  and  technical  skills  of  the  TC’s 
educational  community.  Additionally,  this  job  description  needs  to  match  the  TC’s  end-of-year  evaluation 
criteria,  which  should  be  dynamic  as  well.  Both  the  job  description  and  the  end  of  year  evaluation  should  be 
reevaluated  every  year  or  two  and  revised  as  necessary  to  meet  current  needs. 

There  are  several  things  that  need  to  be  taken  into  consideration  before  these  job  descriptions  can  be 
identified.  There  needs  to  be  a clear  vision  within  the  school  of  what  it  means  to  integrate  technology  into  the 
curriculum.  Additionally,  it  needs  to  be  clear  whom  the  TC  primarily  supports  (ie,  teachers,  students, 
administrators,  etc.)  and  their  skill  level. 

The  job  description  of  the  TC  needs  to  take  into  account  the  amount  and  variety  of  technology  that  the 
coordinator  is  responsible  for  and  the  skill  levels  of  the  teachers  and  students.  As  technology  is  added  to  the 
school  system  the  job  description  needs  to  be  modified  to  account  for  the  change.  Finally,  the  source  of  TC’s 
salary  needs  to  be  clear  as  it  may  determine  the  nature  of  the  tasks  that  the  TC  can  perform. 

The  TC’s  job  description  should  be  based  on  the  unique  needs  of  the  educational  community  in  which 
they  serve.  This  means  that  there  are  going  to  be  different  job  descriptions  for  different  TCs.  Thus,  there 
should  be  variance  in  expectations  across  education  communities  in  regards  to  the  responsibilities  of  TCs. 

In  order  to  assist  the  TC  with  meeting  the  expectations  of  their  job  it  is  important  to  schedule  time  into 
the  daily  routine  of  the  TC  when  they  do  not  work  on  anything  but  curricular  tasks.  The  TC  needs  time  when 
they  are  not  “on  call’’  so  that  they  can  be  guaranteed  uninterrupted  time  for  work  with  curriculum  integration. 
This  time  needs  to  be  equivalent  to  “planning  periods’’  for  teachers  and  it  needs  to  be  “advertised’’  amongst  the 
school  community. 

The  job  description  of  the  TC  should  be  widely  disseminated  within  the  educational  community,  as 
well  as,  the  time  periods  that  are  set  aside  for  the  TC  to  work  on  specific  responsibilities.  Clear,  dynamic  job 
descriptions  and  dedicated  time  for  TCs  for  working  on  PD  will  aide  in  effective  and  efficient  use  of  the 
technology. 
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Appendix  A:  Observation  Results 

Excerpt  from  one  District  TCs.  We  are  in  the  computer  lab  and  the  server  just  crashed. 
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1:47.  Machine  that  is  copying  the  CD  to  the  network  stopped  running.  Message  on  the  screen  read  "general 
failure  reading  drive."  Kathy  makes  a phone  call  "this  is  (Kathy)  could  you  tell  (the  media  coordinator)  to 
reboot  the  lab  server?  (Listening)  ok."  She  hangs  up  the  phone.  Kathy  turns  off  the  remainder  of  the  machines. 
Stating  "this  is  so  aggravating.  I’ll  do  this  another  day.  They’re  locked  up  totally." 

Technician  (T):  I tell  you  what— once  they  get  the  server  restarted  we  could  try  again. 

Kathy  (K):  I’ve  got  a meeting,  I can't. 

T:  Well,  just  get  it  started  and  I’ll  check  on  it  later. 

K:  OK...  Well  let’s  just  wait  the  server’s  been  crashing  a lot  lately.  The  service  person’s  coming  out  to 
work  on  the  problem.  We’ve  at  least  got  the  256  colors  working. 

T:  You’ve  got  my  beeper  number.  Technician  leaves. 

1:50.  Phone  rings.  Kathy  picks  up  the  phone.  "This  is  [Kathy]  (listening)  you  are  kidding!  (Listening)  ok! 
(Listening)  we’re  doing  the  Internet  with  the  chamber  of  commerce  tomorrow  morning.  (Listening)  Ok,  I’ll  be 
right  there."  Kathy  looks  at  me  and  says,  "the  server  won't  come  back  up"  Kathy  grabbed  her  box  and  we  left 
the  computer  lab. 

1:57.  We  stopped  in  office  to  drop  off  the  box.  Someone  in  the  office  asks  Kathy  if  she  could  get  a Word 
Processing  manual  from  her.  She  responds  and  then  asks  for  envelopes  to  put  the  materials  in  the  box  in. 

2:00.  We  leave  the  office  and  head  towards  the  media  center  to  check  on  the  server.  She  gives  me  a newsletter 
that  she  has  written  up  and  explains  that  she  has  a lot  of  friends  in  the  district  and  they  are  now  starting  to  ask 
her  to  come  to  their  home  and  assist  them  with  their  equipment.  She  worked  on  three  machines  last  week.  She 
brought  this  up  to  the  administration  and  they  said  that  she  should  start  charging  them,  which  she  did.  She 
flipped  through  some  of  the  mail  that  she  picked  up  at  the  office.  As  we  turn  the  corner  towards  the  media 
center,  the  technician  is  waiting  in  the  hallway  for  us. 

Technician:  The  server  can’t  find  its  name  and  the  hard  drive  is  making  galloping  sounds.  It’s  not  finding 
the  system. 

Kathy:  This  is  serious  isn’t  it? 

Technician:  yep. 

The  technician  then  reboots  the  machine  and  asks  Kathy  to  listen  closely  to  the  hard  drive. 

Kathy:  I’m  gonna  call  AmeriData.  What’s  it  say?  "system  not  mounted."  I’m  gonna  tell  him  this  is 
serious  big  time  serious. 

We  walk  over  to  her  office  that  is  the  next  hall  over.  She  goes  behind  her  desk  and  picks  up  the  phone. 

She  looks  at  me  and  says  "and,  this  is  just  one  building.  This  building  is  a full-time  job  in  itself.  It  really  is!" 

She  then  calls  AmeriData— gets  an  answering  machine  and  leaves  a message,  “this  is  Kathy  from  xxx  that  server 
has  really  crashed.  I’ll  be  in  a meeting  at  guidance  now  you  could  call  there  and  talk  to  me.  From  3:45-5  I’ll  be 
at  xxx  you  could  call  me  there  the  number  is  xxxxx  or  you  could  call  me  at  home  this  evening  the  number  is 
xxxxx.  The  server  can’t  find  its  name  and  it’s  unable  to  mount  the  system.  I need  to  know  if  this  could  be  fixed 
because  I might  have  to  reschedule  a meeting  for  tomorrow  morning.”  She  made  a phone  call  to  another  place 
there  was  no  answer.  Then  she  called  someone  else  and  left  a message  stating  the  problems  with  the  server. 
And,  called  yet  another  person  who  teaches  adult  education  classes  in  the  evening  using  the  computer  lab  and 
left  a message  with  him. 


Excerpt  of  School  Technology  Coordinator  walking  into  his  office 

9:55  As  soon  as  we  walked  in  to  Rich’s  work  area  his  assistant  said  that  we  had  to  go  back  to  the  principal’s 
office.  He  just  called.  So,  we  turned  around  and  we  headed  back  to  the  main  offices. 

9:56  On  our  way  out  a woman  stopped  Rich  asking  for  help.  She  lost  her  E-mail.  He  told  her  he  would  stop 
by  after  he  helped  someone  else. 

9:57  Then  as  we  got  a little  farther  two  other  people  meet  up  with  Rich  in  the  hallway,  stating  that  he  was 
exactly  the  person  they  wanted  to  see.  He  deferred  their  questions  to  his  assistant. 
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Abstract 

This  study  investigated  whether  it  was  more  beneficial  to  provide  the  learners  in  computer-based 
learning  environments  access  to  on-demand  (self-regulated)  help  after  they  committed  an  error  in  problem 
solving  or  for  the  learning  environment  to  externally  regulate  the  presentation  of  instructional  help. 
Furthermore,  two  different  presentational  formats — textual  and  pictorial — of  instructional  prompts  were 
examined.  This  study  was  conducted  with  a computer-based  learning  environment  that  introduced  high  school 
students  without  any  prior  content-specific  knowledge  to  the  principles  of  parallel  and  series  electrical  circuit 
analysis.  We  found  that  textual  prompts  facilitated  problem  solving  statistically  significantly  better  than 
pictorial  prompts.  Moreover,  the  learners  provided  with  externally  regulated  prompts  reported  statistically 
significantly  more  positive  attitudes  toward  the  prompts  than  learners  in  the  self-regulated  conditions.  Finally, 
the  continuing  motivation  was  statistically  significantly  stronger  in  learners  who  viewed  textual  prompts  than  in 
their  counterparts  in  the  pictorial  prompt  groups. 

Introduction  and  Related  Work 

The  computer-based  instruction  of  electrical  circuit  analysis  techniques  has  received  a significant 
amount  of  interest  over  the  last  fifteen  years,  (see  for  instance  Coulon,  Forte  & Rivera  , 1993;  Hanrahan  & 
Caetano,  1989;  Yoshikawa,  Shintani  & Ohba,  1992).  A wide  variety  of  computer-based  instruction  and  tutoring 
systems  with  the  aim  to  teach  circuit  analysis  techniques  and  to  provide  opportunities  for  practicing  circuit 
analysis  have  been  developed  and  evaluated.  Many  of  the  developed  systems  interact  with  the  learner  to  aid  in 
imparting  the  knowledge  of  the  circuit  analysis  techniques  and  to  provide  feedback  on  learner  input  to  practice 
problems.  In  the  case  of  incorrect  solutions  the  feedback  is  oftentimes  accompanied  by  instructional  prompts 
(help).  These  learner-program  interactions  are  in  the  form  of  text  and/or  graphics  and  are  controlled  (presented) 
by  the  learner  or  the  system.  The  impact  of  both  the  format  and  the  control  (presentation)  of  the  instructional 
prompts  in  circuit  analysis  tutoring  systems  have  not  been  previously  examined  in  detail.  This  study  extends  the 
existing  literature  on  computer-based  instruction  of  electrical  circuit  analysis  in  that  it  examines  the  impact  of 
the  presentation  and  the  format  of  the  instructional  prompts  in  electrical  circuit  tutoring  systems  . 

Schnackenberg,  Sullivan,  Leader,  and  Jones,  (1998)  provide  an  extensive  overview  of  different  studies 
on  the  control  approaches  up  to  their  time  of  writing.  More  recently.  Brown  (2001)  found  that  leaving  the 
navigation  of  computer-based  training  to  the  learner  tends  to  interfere  with  the  instructional  integrity  of  the 
learning  environment,  which  is  counterproductive.  Schnackenberg  et  al.  (1998)  found  that  with  learner  control, 
the  learners  tend  to  skip  practice  examples,  which  tends  to  negatively  affect  learning.  In  the  context  of  the 
highly  structured  content  domain  of  the  present  study,  namely  electrical  circuit  analysis  techniques,  it  appears 
that  program  controlled  instructional  design  and  navigation  would  be  beneficial  (Lawless  & Kulikowich,  1998). 
Furthermore,  program  control  might  be  especially  advantageous  for  learners  with  low  levels  of  prior  knowledge 
in  the  domain  area  (Shin,  Schallert  & Savenye,  1994).  Against  this  background,  it  is  reasonable  to  assume  that 
both  subject-matter  novices  as  well  as  technology  neophytes  should  benefit  from  a program-control  design. 

Overall,  the  issue  of  learner  vs.  external  control  has  so  far  been  primarily  investigated  in  the  context  of 
navigating  the  instructional  and/or  practice  material.  We  are  not  aware  of  a study  on  the  impact  of  learner  vs. 
external  control  of  the  provisioning  of  instructional  prompts  within  a given  practice  problem,  which  is  the  focus 
of  this  study. 

How  should  the  highly  structured  content  on  circuit  analysis  techniques  be  presented  to  learners  in 
order  to  foster  their  initial  knowledge  acquisition  and  to  introduce  them  to  structured,  algorithmic  problem 
solving  associated  with  electrical  circuits?  One  theoretical  approach  that  offers  some  guidance  in  this  area  is 
cognitive  load  theory,  which  provides  the  general  guideline  that  the  limited  capacity  of  the  working  memory  has 
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to  be  taken  into  account  when  designing  an  instructional  module.  One  empirically  validated  instructional 
approach  that  is  particularly  well-suited  to  accommodate  the  limited  capacity  of  learners  is  the  use  of  worked- 
out  examples.  Specifically,  research  suggests  that  worked  examples,  consisting  of  problem  formulation, 
individual  solution  steps,  and  final  solution,  foster  initial  learning  of  highly  structured  subjects  such  as  algebra 
in  Sweller  and  Cooper  (1985)  or  statistics  in  Atkinson,  Renkl  and  Merril  (2003). 

Two  important  factors  found  to  influence  the  degree  of  learning  associated  with  worked  examples  are 
their  structure  in  Atkinson,  Derry,  Renkl  and  Wortham  (2000)  and  the  presence  of  self-explanation  activities 
during  example  processing  as  in  Chi  (2000).  One  way  to  classify  the  structure  of  the  worked  examples  in 
interactive  learning  environments  is  by  the  relationship  to  accompanying  practice  problems.  For  instance, 
presenting  a fully  solved  example  followed  by  a practice  problem  that  requires  the  learner  to  independently 
solve  all  problem  subgoals  (steps)  is  called  example -problem  design.  In  contrast,  in  a problem-example  design, 
the  learner  first  encounters  a practice  problem  and  is  subsequently  presented  with  a fully  worked  example.  In 
the  fading  approach,  the  learner  is  initially  presented  with  a fully  worked  example  and  in  the  next  example  all 
but  one  of  the  problem  subgoals  are  worked  out  and  the  learner  is  required  to  independently  solve  (anticipate) 
the  solution  of  the  missing  problem  subgoal.  In  the  subsequent  example  all  but  two  problem  subgoals  are 
worked  out  and  the  learner  is  required  to  anticipate  the  solutions  to  the  two  missing  problem  subgoals,  and  so 
on,  until  the  learner  is  required  to  anticipate  the  solutions  for  all  problem  subgoals  (independent  problem 
solving).  This  fading  design  comes  in  two  types:  forward-fading,  where  the  solution  steps  are  omitted  starting 
with  the  first  problem  subgoal,  and  backward  -fading , where  the  last  solution  step  is  omitted  first,  then  the  last 
two,  and  so  on.  Recent  studies  found  indications  that  fading,  especially  backward  -fading  has  a positive  effect  on 
learning  (Renkl,  Atkinson  & Grosse,  2004).  Backward  fading  is  therefore  employed  throughout  this  study. 

In  summary,  we  note  that  the  usage  of  the  pictorial  and  textual  presentation  formats  of  the  instructional 
content  and  the  implications  for  the  cognitive  load  have  been  evaluated  by  several  research  groups.  The 
influence  of  pictorial  or  textual  instructional  prompts  in  interactive  learning  environments  with  fading, 
however,  has  not  yet  been  studied  in  detail.  It  is  interesting  and  important  to  understand  the  impact  of  pictorial 
vs.  textual  prompts  on  the  learner’s  performance  as  well  as  the  impact  on  the  learner’s  motivation.  At  the  same 
time  it  is  worthwhile  to  study  these  effects  in  conjunction  with  learner  vs.  external  control  (presentation)  of  the 
prompts. 


Study  Methodology 

The  present  study  manipulates  two  independent  variables,  namely  the  presentation  (external  vs.  self 
regulated)  and  format  (pictorial  vs.  textual)  of  instructional  prompts.  The  study  addresses  the  following  research 
questions: 

• What  is  the  effect  of  the  different  presentation  and  format  of  instructional  prompts  on  the  learner’s 
performance? 

• What  are  the  attitudes  of  the  learners  toward  the  different  types  of  presentation  and  format  of  the 
instructional  prompts? 

Participants  and  Design 

The  participants  in  this  study  were  51  students  from  a small  charter  high  school  in  the  Southwest.  The 
participants  were  recruited  from  a regularly  scheduled  computer  class.  The  experimental  sample  consisted  of  26 
females  and  25  males.  The  grade  level  of  the  participants  ranged  from  eight  to  twelve  (2  eighth-graders,  8 ninth- 
graders,  15  tenth-graders,  18  eleventh-graders,  and  8 twelfth-graders).  The  participants  had  a mean  grade  point 
average  of  3.02  (SD  = .84)  and  had  not  been  exposed  to  formal  instruction  on  electrical  circuit  analysis 
techniques  before  participating  in  this  study. 

Participants  were  randomly  assigned  to  one  of  the  four  experimental  conditions  as  defined  by  a 2 x 2 
factorial  design  with  presentation  (external  vs.  self  regulated)  and  format  (pictorial  vs.  textual)  of  instructional 
prompts  as  factors.  The  resulting  conditions  were:  (1)  self-regulated  textual  prompts,  where  participants  could 
exercise  control  over  the  use  of  text  -based  instructional  prompts  (textual  descriptions),  (2)  externally  regulated 
textual  prompts,  where  the  computer  program  automatically  provided  textual  prompts,  (3)  self-regulated 
pictorial  prompts,  where  participants  could  exercise  control  over  the  use  of  prompts  (diagrams),  and  (4) 
externally  regulated  pictorial  prompts,  where  the  computer  program  automatically  presented  pictorial  prompts. 
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Pencil-Paper  Materials 

The  participants  were  administered  a set  of  pencil-paper  materials  consisting  of  a demographic 
questionnaire,  a pretest,  an  overview  of  parallel  and  series  electrical  circuits,  a posttest,  and  an  attitude  survey. 

Demographic  Questionnaire 

The  questionnaire  collected  basic  demographic  data  (grade  level,  gender,  ethnicity),  as  well  as  the  participants’ 
GPA  and  standardized  test  scores  (Arizona  Instrument  to  Measure  Standards  (AIMS)  or  Stanford  9 math  and 
reading  scores).  The  questionnaire  also  asked  the  participants  whether  they  had  ever  learned  about  electrical 
circuit  analysis  before. 

Pretest 

The  pretest  was  designed  to  assess  the  participants’  prior  knowledge  in  the  area  of  electrical  circuit 
analysis.  It  was  composed  of  six  multiple -choice  questions  relating  to  the  basic  physical  meaning  of  electrical 
current,  voltage,  and  resistance  and  elementary  properties  of  electrical  circuits.  The  participants  could  select 
from  four  response  choices  for  each  question. 

Introductory  Overview 

The  four-page  overview  of  parallel  and  series  electrical  circuits  introduced  the  participants  to  (i)  the 
physical  meaning  and  units  of  electrical  current  and  voltage,  (ii)  electrical  circuit  elements  and  their  graphical 
representations,  such  as  light  bulbs  and  batteries,  and  the  way  circuit  elements  are  connected  with  wires  in  the 
two  main  forms  of  electrical  circuits,  namely  parallel  and  series  circuits,  (iii)  the  physical  meaning  and  units  of 
resistance  as  well  as  Ohm’s  Law,  (iv)  the  calculation  of  the  resistance  of  a parallel  circuit,  and  (v)  the 
calculation  of  the  resistance  in  a series  circuit.  These  last  two  sections  on  calculating  the  resistance  of  series  and 
parallel  circuits  were  not  focused  on  deriving  the  formulas  for  calculating  the  total  resistance  of  the  circuit  from 

the  resistance  values  of  the  individual  circuit  elements  (i.e.,  Rlot  = R]  + Rn  + ...  for  series  circuit  and 
1 / Rlot  = 1/ /?]  + 1 / R0  + ...  for  parallel  circuit). 

The  instructional  goal  was  not  to  teach  the  participants  to  use  these  formulas.  Instead  the  participants 
were  taught  to  calculate  the  total  resistance  from  basic  principles,  namely  Ohm’s  Law  and  the  properties  of 
current  and  voltage  in  the  electrical  circuits.  In  particular,  for  the  series  circuit,  the  participants  were  presented 
with  the  resistance  values  of  the  individual  resistors  in  the  circuit  and  with  the  value  of  the  current  emitted  by 
the  battery  into  the  circuit.  The  participants  were  then  instructed  to  proceed  in  the  following  three  steps  in  the 
calculation  of  the  total  resistance  of  the  series  circuit.  First,  the  participants  studied  that  the  current  flowing 
through  each  of  the  circuit  elements  is  equal  to  the  current  emitted  by  the  battery  and  the  calculation  of  the 
voltage  over  each  individual  resistor  is  done  using  Ohm’s  Law.  Second,  the  participants  were  shown  examples 
where  the  calculation  of  the  total  voltage  over  the  series  arrangement  of  resistors  is  carried  out  by  summing  up 
the  voltages  of  the  individual  resistors.  Third,  the  examples  presented  the  calculation  of  the  total  resistance  of 
the  series  circuit  by  applying  Ohm’s  Law  to  the  entire  circuit,  i.e.,  the  calculation  of  the  total  resistance  of  the 
series  circuit  as  the  sum  of  the  voltages  determined  in  step  2 divided  by  the  current  emitted  by  the  battery.  For 
the  parallel  circuit,  an  analogous  solution  strategy  was  presented. 

Posttest 

The  posttest  contained  eight  complex  problems,  more  specifically,  four  problems  (two  for  each  type  of 
the  electrical  circuits,  parallel  and  series)  to  measure  the  performance  on  near  transfer  and  four  problems  (two 
for  each  type  of  the  electrical  circuits,  parallel  and  series)  to  assess  the  far-transfer  learning. 

The  near-transfer  problems  had  the  same  underlying  structure  but  different  surface  characteristics  from 
the  practice  problems  encountered  during  the  learning  (computer)  phase.  They  required  the  participants  to 
perform  the  same  tasks  (e.g.,  calculating  the  individual  voltage  or  current  respectively,  determining  the  total 
voltage  or  current  respectively,  and  finally  computing  the  total  resistance)  as  they  learned  in  the  computer-based 
module.  Despite  having  the  same  structure  and  requiring  the  same  solution  steps  as  the  practice  problems  from 
the  learning  phase,  the  near-transfer  problems  appeared  different  since  they  had  different  cover  stories  and 
current,  voltage,  and  resistance  values. 

The  far-transfer  problems  had  different  underlying  structure  and  surface  features  as  compared  to  the 
computer-based  practice  problems.  In  particular,  in  the  far-transfer  series  circuit  problems  the  participants  were 
given  the  individual  resistance  values  and  the  voltage  over  one  of  the  resistors.  The  far-transfer  series  circuit 
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problem  asked  the  participants  to  calculate  the  battery  voltage.  To  solve  this  problem,  the  participants  had  first 
to  use  Ohm’s  Law  to  calculate  the  current  in  the  series  circuit  from  the  resistance  value  of  the  one  resistor  for 
which  the  voltage  was  given.  The  participants  then  had  to  notice  that  the  current  is  the  same  in  all  resistors  and 
had  to  calculate  the  voltages  over  the  other  resistors  in  the  circuit  from  the  current  determined  in  the  first  step 
and  the  values  of  the  individual  resistors  using  again  Ohm’s  Law.  In  the  third  and  final  solution  step,  the 
participants  had  to  sum  up  the  voltages  over  the  individual  resistors  to  obtain  the  total  voltage  (battery  voltage) 
over  the  circuit.  The  far-transfer  parallel  circuit  problems  were  structured  analogously. 

Attitude  Survey 

A 14-item  attitude  survey  was  used  to  collect  data  on  participant  attitudes  and  motivation.  The  survey 
pertained  to  the  overall  effectiveness  of  the  computer-based  program,  the  format  of  the  instructional  prompts, 
and  participant  continuing  motivation.  The  individual  items  were  five-choice  Likert-type  questions.  The 
response  choices  were  assigned  ratings  of  strongly  agree,  agree,  neither  agree  nor  disagree,  disagree,  and 
strongly  disagree.  The  attitude  items  were  grouped  into  three  categories,  namely  instructional  effectiveness  (7 
items),  role  of  instructional  prompts  (4  items),  and  continuing  motivation  (3  items). 

Computer-Based  Learning  Environment 

The  module  was  developed  using  Director  MX  (by  Macromedia,  Inc.  (2002))  software,  which  is  an 
authoring  tool  for  creating  rich  multimedia  programs.  The  module  was  programmed  to  operate  in  one  of  four 
modes  that  corresponded  to  the  four  experimental  conditions  of  the  current  study. 

The  goal  of  the  computer-based  learning  environment  was  to  deliver  instruction  on  the  principles  of 
calculating  resistance  in  parallel  and  series  electrical  circuits.  The  aim  of  the  program  was  to  present  worked 
(solved)  examples  to  the  participants  and  to  scaffold  their  learning  by  progressively  reducing  the  number  of 
worked  solution  steps  and  increasing  the  amount  of  independent  problem  solving  by  the  participants.  The 
environment  presented  two  sets  (parallel  and  series)  with  four  problems  each,  constituting  a total  of  eight 
problems.  The  following  is  the  cover  story  from  one  of  the  instructional  examples  that  were  shown  to  the 
participants  on  the  computer  screens  during  the  learning  phase: 

To  operate  an  aquarium  you  wire  the  pump  with  a resistance  of  R = 20  Q.  and  the  aquarium  light 

with  a resistance  of  Rj  = 40  Q.  in  parallel.  You  connect  this  parallel  circuit  to  a battery  with  a voltage 

of  V,  = 5 V.  What  is  the  total  resistance  Rtot  of  this  parallel  circuit? 

Each  problem  had  exactly  three  solution  steps.  Each  step  was  clearly  labeled  and  visually  distinguished 
from  the  other  steps.  The  computer  module  revealed  one  step  at  a time  after  the  participants  clicked  the  “Next” 
button,  thus  allowing  the  participants  to  control  the  pace  of  their  learning.  The  participants  proceeded  through 
the  module  by  clicking  on  the  “Next  Problem”  buttons  after  inspecting  all  three  steps  in  each  problem.  The 
navigation  was  linear  and  the  participants  could  not  return  to  previous  steps  and  problems  once  they  finalized 
their  answers. 

The  first  problem  in  each  of  the  sets  of  four  problems  was  fully  solved  (worked),  whereas  in  the 
subsequent  problems  the  worked  steps  were  backward  faded  and  the  participants  had  to  anticipate  the  correct 
solution  to  the  missing  steps.  Specifically,  the  participants  had  to  independently  solve  one  step  (the  last  one)  in 
the  second  problem  of  each  set,  two  steps  (the  last  two)  in  the  third  problem  of  each  set,  and  were  responsible 
for  independently  solving  all  three  steps  in  the  last  problem  of  each  set. 

In  the  case  of  incorrect  anticipation,  the  computer-based  learning  environment  offered  an  instructional 
prompt  that  was  either  externally  regulated  or  requested  by  the  participant,  depending  on  the  treatment 
condition.  Participants  in  the  externally  regulated  groups  were  always  presented  with  the  instructional  prompt  if 
they  made  a mistake  while  solving  the  individual  steps.  On  the  other  hand,  the  decision  to  view  the  instructional 
prompts  was  solely  at  the  discretion  of  the  participants  in  the  self-regulated  conditions.  They  were  offered  the 
option  to  receive  the  instructional  prompt  but  could  refuse  the  help. 

The  instructional  prompts  were  presented  in  two  different  formats,  depending  on  the  treatment 
condition.  In  the  textual-based  prompt  groups,  the  prompts  were  verbal  reminders  of  Ohm’s  Law  and  the 
properties  of  currents  and  voltages  in  series  and  parallel  circuits.  These  reminders  were  tailored  to  the  individual 
problem  steps.  The  pictorial-based  prompts  were  presented  as  drawings  illustrating  the  current  flow  and 
voltages  in  series  and  parallel  circuits,  as  well  as  Ohm’s  Law  tailored  to  the  individual  problem  step. 

Once  the  request  for  the  instructional  prompt  was  entered,  the  prompt  appeared  on  the  screen  next  to 
the  solution  step  that  needed  to  be  solved.  The  participants  were  given  two  attempts  at  solving  each  missing 
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step.  The  correct  solution  was  then  displayed  on  the  screen.  The  solved  steps  remained  visible  on  the  screen 
after  the  final  answer  was  presented,  allowing  the  participants  to  study  the  entire  solution  (worked  example). 


Procedure 

Groups  of  8 to  15  participants  attended  one  of  the  five  scheduled  experimental  sessions.  The  average 
duration  of  each  session  was  approximately  60  minutes.  The  participants  took  part  in  the  study  in  a computer 
lab  in  their  high  school.  Each  participant  was  seated  in  front  of  a Windows-based  desktop  computer.  The 
experimenter  instructed  the  participants  to  work  independently  of  their  peers.  The  participants  first  filled  in  the 
demographic  questionnaire.  Next,  they  answered  the  pretest.  The  participants  proceeded  to  study  the 
introductory  overview  on  electrical  circuits.  After  studying  the  introductory  instructional  text  the  participants 
worked  through  the  problems  in  the  computer-based  learning  environment.  During  this  phase  the  experimental 
variation  took  place.  Immediately  after  completing  the  computer-based  instructional  program  the  participants 
were  administered  the  posttest.  Finally,  they  indicated  their  responses  on  the  attitude  survey.  Each  participant 
completed  this  entire  procedure  in  one  session. 

Scoring 

The  participants’  performance  on  the  pretest,  practice  problem  solving  during  the  computer-based 
instruction,  and  the  posttest  (near-  and  far-transfer  problems)  as  well  as  their  responses  to  the  attitude  survey 
were  scored.  The  computer-based  learning  module  automatically  recorded  the  en  route  practice  (accuracy  of 
solving  the  missing  steps)  and  time  on  the  computer.  The  maximum  pretest  score  was  6,  one  point  for  each 
correctly  answered  multiple -choice  question.  There  were  a total  of  12  unsolved  steps  in  the  computer-based 
learning  environment.  The  participants  were  given  two  attempts  at  solving  each  of  the  12  unsolved  steps.  For 
each  correctly  solved  step,  one  point  was  awarded,  thus  producing  a maximum  score  of  12  for  each  of  the 
solving  attempts,  i.e.,  the  first  and  second  anticipations.  (A  score  of  zero  was  assigned  for  the  second 
anticipation  if  the  first  anticipation  was  correct.)  The  individual  scores  for  each  of  the  anticipations  were 
summed  and  divided  by  12  in  order  to  obtain  the  proportion  of  problem  steps  that  were  correctly  solved  on  the 
first/second  anticipation.  The  values  for  the  first  and  second  anticipations  ranged  from  0 to  1.  The  eight  posttest 
problems  had  three  distinctive  solution  steps  each,  thus  resulting  in  a maximum  score  of  three  points  for  each 
problem,  equaling  to  a maximum  total  score  of  24  (12  points  each  were  associated  with  the  performance  on  the 
near-  and  far-transfer  problems,  respectively).  On  the  attitude  survey,  a rating  of  strongly  agree  received  a score 
of  5,  agree  a score  of  4,  a rating  of  neither  agree  or  disagree  equaled  to  3,  disagree  was  scored  as  2,  and  a rating 
of  strongly  disagree  received  a score  of  1 . 

Data  Analysis 

There  was  no  significant  difference  in  the  pretest  scores  between  the  participants  randomly  assigned  to 
the  textual  and  pictorial  prompts  groups  F(l,47)  = 1.70,  p = .20.  Similarly  there  was  no  significant  difference  in 
the  pretest  scores  between  the  participants  randomly  assigned  to  the  self -regulated  and  externally  regulated 
prompts  groups  F = .01,  p = .94.  The  near  transfer  and  far  transfer  posttest  scores  were  analyzed  using  2 
(Prompt  Format:  Textual  and  Pictorial  format)  x 2 (Prompt  Presentation:  Self-regulated  or  Externally  regulated) 
x 2 (Posttest  Problem  Type:  Near-transfer  and  Far-transfer  posttest  scores)  analysis  of  variance  (ANOVA). 
Prompt  format  and  prompt  presentation  were  between-subjects  variables  in  the  analysis  and  posttest  problem 
type  was  within-subjects  variable.  En  route  data  on  performance  on  practice  problems  in  the  computer-based 
learning  environment  was  also  collected  and  analyzed.  Attitude  data  were  analyzed  using  2 (textual  or  pictorial 
format  of  prompts)  x 2 (self  or  external  presentation  of  prompts)  x K (number  of  attitude  items  for  each  of  the 
three  item  categories:  overall  effectiveness  of  instruction  (K=7),  effectiveness  of  prompts  (k=4),  and  continuing 
motivation  k=3)  multivariate  analyses  of  variance  (MANOVA). 

Results 


Achievement 

The  unadjusted  mean  scores  and  standard  deviations  for  each  treatment  group  on  the  near-  and  far- 
transfer  posttest  problems  are  shown  in  Table  1. 


731 


Table  1:  Posttest  scores  by  format  and  presentation  of  prompts. 


Format  of  PromDts 

Presentation  of  Prompts 

Textual 

Pictorial 

Self 

External 

(A  = 24) 

(A  = 27) 

QL 

= 24) 

(A=  27) 

Measure 

M SD 

M SD 

M 

SD 

M SD 

Near  Transfer 

9.08*  2.29 

6.89  3.70 

8.54 

3.24 

131  3.26 

Far  Transfer 

1.04  1.40 

1.00  1.18 

1.29 

1.43 

.78  1.09 

*p=m 


An  ANOVA  revealed  that  there  was  no  significant  difference  between  the  two  levels  of  the 
presentation  factor  (self  vs.  external  regulation)  on  the  overall  posttest  total,  F ratio  F(l,47)  = 3.43,  mean  square 
error  MSE  = 11.20,  significance  level  p = .07.  There  was  a statistically  significant  difference  when  comparing 
the  two  different  formats  of  prompts  (pictorial  vs.  textual).  Specifically,  on  the  near-transfer  posttest, 
participants  presented  with  text  -based  instructional  prompts  scored  significantly  higher,  achieving  a mean  score 
of  9.08  (76%),  than  their  counterparts  provided  with  pictorial-based  prompts,  who  achieved  a mean  score  of 
6.89  (57%  ).  This  difference  on  the  near  transfer  posttest  scores  was  statistically  significant  Ff  1,47)  = 6.50,  MSE 
= 9.69,  p = .01.  Cohen’s /statistic  for  these  data  yields  an  effect  size  estimate  of  .37  for  the  near-transfer 
posttest  problems,  which  approaches  a large  effect.  This  advantage  did  not,  however,  extend  to  the  performance 
on  the  far-transfer  items.  Finally,  there  was  no  significant  interaction  between  the  two  factors. 

Practice 

The  performance  on  en  route  practice  problems  was  analyzed  using  2 (textual  or  pictorial  format  of 
prompts)  x 2 (self  or  external  presentation  of  prompts)  analysis  of  variance  (ANOVA).  The  proportions 
(accuracy)  of  correctly  solving  the  unsolved  practice  problem  steps  at  the  first  and  second  attempt  are  presented 
in  Table  2. 


Table  2:  Accuracy  of  practice  problem  solving  by  format  and  presentation  on  prompts 


Format  of  Prompts 

Presentation  of  Prompts 

Textual 

Pictorial 

Self 

External 

(A  = 24) 

(A  = 27) 

QL 

= 24) 

(A  =27) 

Measure 

M SD 

M SD 

M 

SD 

M SD 

First  Anticipation 

\ o 

b 

00 

.82  .13 

.86 

.14 

.88  .11 

Second  Anticipation 

.05  .07 

.13*  .09 

.09 

.09 

.09  .08 

*p<.0l 


The  participants  in  the  textual  prompts  groups  solved  on  average  93%  of  the  unsolved  problems  steps, 
i.e.,  11.16  out  of  the  12  unsolved  steps,  at  the  first  attempt,  whereas  the  participants  in  the  pictorial  prompts 
group  solved  82%  of  the  unsolved  steps  at  the  first  attempt.  The  ANOVA  revealed  that  this  difference 
corresponds  to  a significant  main  effect  on  first  anticipation  for  format  of  prompts,  FIT, 47)  = 12.00,  MSE  = .01, 
p<  .01.  The  differences  on  accuracy  of  anticipations  on  the  first  anticipation  between  the  self  and  external 
approach  to  presentation  of  prompts  were  non-significant  as  was  the  interaction  between  presentation  and 
format  of  instructional  prompts.  There  was  a significant  main  effect  for  format  of  prompts  on  the  second  trial  of 
solving  practice  problems.  In  particular,  the  participants  who  received  pictorial-based  prompts  had  a 
significantly  higher  probability  of  accurately  solving  the  practice  problems  on  the  second  trial  as  compared  to 
participants  who  were  assigned  to  the  text-based  prompts  groups,  F(l,47)  = 12.77,  MSE  = .01,  p<  .01. 

Table  3:  Participant  attitude  scores  by  format  and  presentation  of  prompts 


Format  of  Prompts  Presentation  of  Prompts 

Textual  Pictorial  Self  External 

(A  = 24)  (A  = 27)  (A  = 24)  (A  = 27) 

M M F M M F 
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Attitude  Category 


Instructional  Effectiveness 

4.05 

3.78 

3.18 

3.91 

3.92 

.01 

Role  of  Instructional  Prompts 

4.01 

3.93 

.19 

3.73 

4.18 

5.40* 

Continuing  Motivation 

3.83 

3.36 

6.36* 

3.68 

3.50 

1.06 

Role  of  Instr.  Prompts  Items 

The  instructional  explanations  (hints) 

helped  me  to  learn 

4.04 

4.11 

.07 

3.75 

4.37 

6.18* 

Text -based  instructional  explanations  (hints) 
were/would  be  helpful 

3.83 

3.37 

2.40 

3.29 

3.85 

3.60 

Pictorial-based  instructional  explanations 
(hints) 

were/would  be  helpful 

4.21 

4.19 

.01 

4.13 

4.26 

.35 

Having  control  over  the  instructional 
explanations 

(hints)  was/would  be  beneficial  for  my 
learning 

4.21 

4.04 

.52 

4.00 

4.22 

.89 

Continuine  Motivation  Items 

I would  like  to  learn  more  about  electrical 

circuits 

3.79 

2.93 

12.34** 

3.54 

3.15 

2.13 

After  completing  the  instructional  unit,  I find 
science  more  interesting 

3.63 

3.30 

1.91 

3.54 

3.37 

.50 

I would  recommend  this  instructional  unit  to 

4.08 

3.85 

.81 

3.96 

3.96 

.00 

others 

p < .05,  * p < ■ 01 

Instructional  Time 

In  order  to  test  if  the  amount  of  time  participants  spent  on  acquiring  initial  knowledge  during  the 
paper-based  training  and  learning  in  the  computer-based  learning  environment  influenced  their  performance  on 
the  posttest,  an  analysis  of  variance  (ANOVA)  was  conducted  on  these  times.  No  statistically  significant 
differences  between  the  different  treatment  groups  were  found. 

Attitudes 

The  attitude  items  were  scored  on  a five-point  scale,  ranging  from  5 indicating  strong  agreement  with 
the  positive  statements  to  1 corresponding  to  strong  disagreement.  The  overall  mean  score  across  all  the  14 
attitude  items  on  the  survey  for  all  participants  was  3.86  ( SD  = .49). 

The  attitude  items  were  grouped  into  three  categories,  namely  instructional  effectiveness  (7  items),  role 
of  instructional  prompts  (4  items),  and  continuing  motivation  (3  items).  The  mean  attitude  scores  by  format  and 
presentation  of  prompts  for  participant  responses  on  the  three  main  categories  of  attitude  items  on  the  five-point 
attitude  survey  are  presented  in  Table  3.  Multivariate  analyses  of  variance  (MANOVA)  of  the  attitude  items 
related  to  the  role  of  instructional  prompts  revealed  that  there  was  a significant  overall  main  effect  across  the 
four  items  relating  to  the  presentation  of  prompts,  (M  = 4.18  for  externally  regulated  prompts  and  M = 3.73  for 
self-regulated  prompts),  F(1 ,47)  = 5.40,  MSE  = .45,  p = .03.  Cohen’s /statistic  for  these  data  yields  an  effect 
size  of  .34,  which  corresponds  to  medium  to  large  effect.  In  addition,  a significant  overall  main  effect  was 
discovered  across  the  items  relating  to  continuing  motivation  for  the  format  of  prompts  (M  = 3.83  for  text  -based 
prompts  and  3.36  for  pictorial-based  prompts),  F(  1,47)  = 6.36,  MSE  = .45,  p = .02.  Cohen’s/for  these  data 
yields  an  effect  size  of  .37,  which  corresponds  to  medium  to  large  effect.  The  differences  for  the  format  and 
presentation  of  prompts  on  the  attitude  survey  items  relating  to  the  instructional  effectiveness  were  non- 
significant. 

Univariate  analysis  of  variance  on  the  four  items  relating  to  the  effectiveness  of  the  instructional 
prompts  by  presentation  of  the  prompts  revealed  significantly  higher  scores  on  the  item  “The  instructional 
explanations  (hints)  helped  me  to  learn’’  for  the  externally  regulated  prompts  (M  = 4.37  for  externally  regulated 
prompts  and  3.75  for  self -regulated  prompts,  F{  1,49)  = 6.18,  MSE  = .79,  p = .02). 

Univariate  analysis  of  variance  on  the  three  items  relating  to  the  continuing  motivation  by  format  of 
the  prompts  revealed  significantly  higher  scores  on  the  item  “I  would  like  to  learn  more  about  electrical 
circuits’’  for  the  textual  prompts  (M  = 3.79  for  text -based  prompts  and  2.93  for  pictorial  prompts,  F(  1,49)  = 
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12.34,  MSE  = .77,  p < .01). 


Discussion 

The  two  main  research  questions  addressed  in  the  present  study  focused  on  the  impact  of  the  format 
(textual  or  pictorial)  and  presentation  (self  or  externally  regulated)  of  instructional  prompts  on  the  learners’ 
performance  and  attitudes.  Significant  differences  were  revealed  for  the  accuracy  of  anticipations  on  practice 
problems  during  the  learning  phase  in  the  computer-based  environment.  In  particular,  the  learners  assigned  to 
the  textual-based  prompt  groups  were  significantly  more  successful  in  correctly  solving  the  individual  solution 
steps  at  the  first  problem-solving  attempt  than  their  counterparts  in  the  pictorial-based  prompt  groups.  This 
finding  corresponds  to  a large  effect  and  is  therefore  of  practical  significance.  The  learners  who  were  assigned 
to  the  treatment  conditions  with  pictorial  prompts,  on  the  other  hand,  had  a significantly  higher  success  rate  at 
the  second  anticipation  compared  to  learners  in  the  text -based  prompt  conditions.  One  way  to  account  for  the 
higher  success  rate  of  the  learners  with  the  pictorial  prompts  in  the  second  attempt  is  that  these  learners  had  a 
significantly  higher  probability  of  advancing  to  the  prompt  and  second  trial  because  of  their  higher  error  rates  at 
the  first  anticipation.  In  particular,  for  18%  of  the  solution  steps  the  learners  in  the  pictorial-based  prompt  group 
advanced  to  the  prompt  and  second  trial,  compared  to  7%  for  the  learners  in  the  text  -based  prompt  group. 

We  also  found  that  the  textual  prompt  format  leads  to  significantly  higher  near-transfer  posttest 
performance  compared  to  the  pictorial  prompt  format.  The  advantage  of  textual  prompts  over  pictorial  prompts 
on  the  near-transfer  learning  yielded  a large  effect,  which  indicates  this  is  also  of  practical  relevance.  The 
analysis  indicates  that  the  advantage  of  the  textual  format  of  instructional  prompts  cannot  be  attributed  to  the 
amount  of  instructional  time.  The  significantly  better  performance  with  the  textual  prompts  indicates  that  the 
textual  representation  of  the  electrical  analysis  techniques  is  more  suitable  for  learners  without  prior  exposure  to 
electrical  circuit  analysis.  The  results  suggest  that  all  the  learners,  regardless  of  the  treatment  condition, 
encountered  difficulties  when  attempting  to  solve  the  far-transfer  problems.  Essentially,  we  encountered  a floor 
effect  on  this  measure. 

The  results  from  the  attitude  survey  indicate  that  the  learners  in  the  text -based  group  expressed 
significantly  more  positive  attitudes  towards  the  statements  relating  to  continuing  motivation.  In  particular  the 
learners  that  had  experienced  text -based  prompts  had  a significantly  more  positive  attitude  toward  the  survey 
item  relating  directly  to  future  study  of  electrical  circuit  analysis.  This  more  positive  attitude  is  consistent  with 
the  higher  posttest  scores  of  the  learners  in  the  text  -based  prompt  group.  Indeed,  the  higher  posttest  scores 
suggest  that  these  learners  had  acquired  a better  mastery  of  the  instructional  material,  which  may  have  made 
them  more  confident  about  their  newly  acquired  skills,  and  had  resulted  in  higher  motivation  for  further  study  in 
the  content  area  of  electrical  circuits.  This  difference  in  attitudes  towards  learning  more  about  electrical  circuits 
corresponds  to  a large  effect,  indicating  that  this  difference  has  practical  relevance. 

The  results  for  the  learner  attitudes  toward  the  statements  relating  to  the  role  of  the  instructional 
prompts  indicate  that  the  external  regulation  of  the  prompts  is  perceived  as  significantly  more  appealing  than 
the  self-regulation  of  the  prompts.  This  is  an  interesting  result  considering  that  there  were  no  significant 
differences  for  external  vs.  self  regulation  of  the  prompts  in  terms  of  the  performance  on  the  posttest  and  the 
instructional  time.  The  learners  with  the  low  level  of  prior  knowledge  may  have  preferred  that  the  prompts  were 
automatically  presented  by  the  system  instead  of  being  forced  to  decide  for  themselves  whether  or  not  they 
should  view  them.  Overall,  the  results  of  the  attitude  survey  suggest  that  employing  text -based  prompts  and 
having  the  prompts  under  the  control  of  the  instructional  module  are  preferred  by  students  in  a module  on 
electrical  circuit  analysis  for  high  school  students  with  little  knowledge  in  this  domain. 

There  are  several  interesting  avenues  to  pursue  in  future  research  on  computer-based  interactive 
learning  modules  with  instructional  prompts.  One  avenue  is  to  investigate  the  impact  of  text  vs.  pictorial 
prompts  on  learners  with  a higher  level  of  prior  knowledge  of  general  engineering  analysis  techniques,  such  as 
college  freshmen  or  sophomore  engineering  students.  Another  avenue  is  to  study  the  impact  of  more  complex 
and  elaborate  pictorial  prompts  that  are  designed  to  foster  the  acquisition  of  the  more  expert -like  graphical 
representation  common  in  electrical  circuit  analysis. 
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Validating  Instructional  Design  and  Development  Models 

Rita  C.  Richey 
Wayne  State  University 

Abstract 

Many  recognize  that  instructional  design  and  development  (ID)  models  should  be  substantiated  by 
systematic  validation  rather  than  relying  primarily  on  user  testimonials  as  evidence  of  their  effectiveness 
(Gustafson  & Branch , 2002).  Model  validation  projects,  however,  seldom  become  a priority.  The  paucity  of 
such  efforts  may  be  more  a reflection  of  time  constraints  and  ill-defined  model  validation  procedures,  rather 
than  a lack  of  appreciation  of  the  fundamental  need  for  validation.  This  paper  describes  the  general  nature  of 
ID  models  and  ID  model  validation,  and  then  explores  five  alternative  approaches  to  the  validation  process. 

The  Nature  of  ID  Models 

Models,  by  definition,  are  simplified  representations,  and  they  are  often  idealized.  Nonetheless,  models 
provide  structure  and  order  to  complex  real  life  events  that  on  the  surface  can  seem  chaotic.  ID  models  are  no 
different.  In  most  cases,  the  use  of  an  ID  model  calls  for  considerable  interpretation  and  amplification  to 
provide  the  detail  required  for  specific  applications. 

Most  ID  models  are  visual  diagrams  and  are  procedural  in  nature.  Notable  examples  are  the  generic 
flowchart  models  of  Dick,  Carey  and  Carey  (2001)  and  Smith  and  Ragan  (1998).  Other  visual  formats  are  also 
used,  such  as  the  embedded  circle  design  of  the  Morrison,  Ross  and  Kemp  (2003)  Model.  The  majority  of  these 
procedural  models  pertain  to  comprehensive  design  projects.  Gustafson  and  Branch  (2002)  have  described  these 
models  as  beginning  with  various  forms  of  analysis,  and  progressing  through  the  design  of  a set  of 
specifications  for  the  learning  environment  and  the  development  of  learning  materials.  Evaluation  activities 
permeate  the  entire  process,  even  through  the  management  of  the  ongoing  implementation  of  the  products. 

There  are  many  variations  of  this  general  ID  process  often  represented  by  more  specific  models  intended  to 
relate  to  the  idiosyncrasies  of  specific  groups  of  learners,  learning  environments,  types  of  delivery  systems,  or 
even  specific  design  philosophies. 

Conceptual  design  models  have  less  consistent  formats  than  procedural  models,  but  there  is  one  large 
segment  of  conceptual  models  that  use  the  taxonomy  format.  Seels  (1997)  has  explored  a range  of  taxonomic 
models  in  our  field.  These  models  began  with  Dale’s  (1946)  Cone  of  Experience,  an  early  media  selection 
model  that  classified  media  on  a concrete-to-abstract  continuum.  Other  conceptual  models  in  the  literature  are 
more  varied  in  format.  Some  path  diagrams  can  also  be  viewed  as  conceptual  ID  models.  These  are  graphical 
displays  of  the  patterns  of  direct  and  indirect  relationships  between  variables.  One  such  example  is  Richey’s 
(1992)  model  of  factors  predicting  employee  training  outcomes.  Another  type  of  conceptual  model  is  Hannafin 
and  Rieber’s  (1989)  ROPES+  rneta- model  for  designing  computer-based  instruction. 

The  General  Nature  of  ID  Model  Validation 

Practically  speaking,  most  designers  seem  to  view  models  as  “valid”  if  they  address  the  needs  and 
constraints  of  their  workplaces,  are  easily  used,  and  if  their  use  tends  to  result  in  products  and  programs  that  are 
well  received  by  one’s  clients.  One’s  own  experiences  or  the  recollections  of  others  serve  as  the  supporting 
data.  Theorists  and  model  developers,  on  the  other  hand,  are  likely  to  assume  the  validity  of  a model  if  it  is  a 
logical,  coherent  entity  with  literature  support.  They  are  also  influenced  by  the  practical  results  of  its  use  and 
user  satisfaction.  Certainly  the  prominent  models  in  the  ID  literature  have  been  used  successfully  for  many 
years.  However,  even  with  these  models,  the  data  supporting  validity  tends  to  be  rare  or  non-existent. 

Here,  I am  viewing  ID  model  validation  as  a carefully  planned  process  of  collecting  and  analyzing 
empirical  data  to  demonstrate  the  effectiveness  of  a model’s  use  in  the  workplace  or  to  provide  support  for  the 
various  components  of  the  model  itself.  Akin  to  the  use  of  the  term  validation  in  relation  to  measurement  and 
research  design,  this  is  a process  that  concerns  the  extent  to  which  inferences  are  appropriate  and  meaningful. 


Internal  and  External  Validation  of  ID  Models 

ID  model  validation  is  viewed  here  in  two  ways  - as  either  internal  validation,  that  is  a validation  of 
the  components  and  processes  of  an  ID  model,  or  external  validation,  that  is  a validation  of  the  impact  of  the 
products  of  model  use.  The  findings  of  all  model  validation  studies  form  critically  needed  parts  of  the 
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instructional  design  knowledge  base. 

The  Character  of  Internal  Model  Validation . Internal  validation  focuses  upon  the  integrity  of  the 
model  and  its  use.  Such  studies  are  typically  conducted  during  model  construction  or  in  it’s  the  early  stages  of 
use.  They  provide  data  to  support  each  component  of  the  model,  as  well  as  the  relationship  between  the 
components  and  the  processes  involved.  In  many  respects,  internal  validation  studies  can  be  seen  as  a type  of 
formative  evaluation  of  the  model.  These  investigations  answer  questions  such  as  the  following: 

Model  Components 

1.  Are  all  steps  included  in  the  model  necessary?  Are  there  any  steps  missing  in  the  model?  Are  there 
any  that  need  to  be  clarified? 

2.  Is  the  sequence  of  steps  appropriate?  Are  the  steps  manageable  in  the  prescribed  sequence? 

3.  To  what  extent  does  the  model  address  those  factors  in  the  instructional,  pre -instructional,  and 
work  environments  that  contribute  to  learning? 

4.  To  what  extent  does  the  model  address  those  factors  in  the  instructional,  pre -instructional,  and 
work  environments  that  contribute  to  transfer  or  performance  improvement? 

Model  Use 

5.  To  what  extent  is  the  model  usable  for  a wide  range  of  design  projects?  Does  it  easily 
accommodate  many  types  of  content,  instructional  products,  delivery  systems,  and  instructional 
strategies? 

6.  To  what  extent  is  the  model  usable  in  a wide  range  of  work  environments  given  varying 
organizational  cultures  and  learner  populations,  as  well  as  varying  resources  and  constraints? 

7.  Can  the  steps  be  reasonably  implemented  by  both  novice  and  expert  designers?  Can  the  model  be 
used  without  assistance  by  a trained  designer? 

8.  Can  the  steps  be  completed  efficiently  under  most  working  conditions? 

9.  Is  the  level  of  client  involvement  in  the  design  and  development  process  appropriate  for  most 
work  settings?  To  what  extent  are  the  clients  satisfied  with  the  design  and  development  process? 

10.  Is  the  use  of  this  model  cost  effective? 

It  would  be  unlikely  that  a particular  validation  study  would  address  each  of  these  concerns,  or  give  equal 
emphasis  to  each  factor.  Nor  is  this  list  presumed  to  be  complete;  other  issues  may  be  pertinent  to  particular 
models  or  particular  users. 

The  Character  of  External  Validation.  External  model  validation  addresses  the  effects  of  using  the 
model  - the  instructional  products  themselves,  and  impact  of  these  products  on  learners,  clients  and 
organizations.  In  many  respects,  these  studies  can  be  seen  as  summative  or  confirmative  evaluations  of  the 
model.  They  address  questions  such  as  the  following: 

Product  Characteristics 

1.  To  what  extent  does  the  resulting  instruction  meet  learner  needs,  client  needs,  and  client 
requirements? 

2.  To  what  extent  is  the  resulting  instruction  motivating  and  interesting  to  the  target  audience? 
Were  the  learners  engaged  in  the  instructional  activities? 

3.  To  what  extent  do  learners  accept  the  resulting  instruction,  its  delivery  system,  and  its 
navigation  techniques  (if  applicable)? 

Impact  of  Instruction 

4.  To  what  extent  do  changes  occur  in  learners’  knowledge,  attitudes,  and/or  behaviors  after 
instruction? 

5.  To  what  extent  are  these  changes  retained  over  time? 

6.  To  what  extent  does  the  instruction  result  in  efficient  learning? 

7.  To  what  extent  do  resulting  behavior  changes  impact  the  organization’s  performance? 

8.  To  what  extent  are  the  clients  satisfied  with  the  instruction  and  its  impact? 

External  validations  can  be  complex  research  undertakings  due  to  the  large  number  of  extraneous  factors  that 
can  influence  the  findings.  Such  findings  may  be  impacted  by  factors  such  as  instructor  characteristics,  learner 
distractions,  past  history  and  organizational  priorities  to  name  just  a few.  Nonetheless,  external  validations 
address  those  factors  that  many  consider  to  be  the  central  focus  of  design  efforts. 

Key  Factors  Impacting  ID  Model  Validation 

ID  model  use,  and  its  subsequent  validation,  is  affected  by  a large  number  of  factors.  Some  of  these 
factors  tend  to  lead  to  variations  in  the  model’s  use  and,  at  times,  they  even  lead  to  variations  in  the  models 
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themselves.  An  important  part  of  any  validation  effort  is  to  identify  those  factors  that  may  be  influencing  the 
use  of  an  ID  model  in  the  target  environment.  It  is  one  function  of  the  validation  research  design  to  control  for 
these  variables.  In  a typical  work  setting  this  is  not  always  easy.  Nonetheless,  there  are  two  factors  which  are 
especially  critical  to  address  - the  context  in  which  the  model  is  used  and  the  expertise  of  the  designer. 

Design  Context  Effects.  There  is  an  implicit  (if  not  an  explicit)  assumption  that  most  of  the  widely 
published  and  taught  models  can  be  used  in  all  design  contexts,  and  there  is  a long  history  of  many  ID  mo  dels 
being  successfully  used  in  a variety  of  settings  - corporate,  educational,  health  care,  and  military,  for  example. 
Most  assume  the  universal  applicability  of  instructional  systems  design  (ISD)  procedures,  and  traditionally  this 
has  been  viewed  as  a major  advantage  of  the  methodology. 

Edmonds,  Branch  and  Mukherjee  (1994)  posit  that  the  success  of  an  ID  model  is  dependent  upon  the 
extent  to  which  a match  between  the  application  context  and  the  context  for  which  the  model  was  originally 
intended.  The  contextual  elements  they  stress  are  not  only  setting,  but  also  differences  in  type  of  content  and  the 
type  of  product  being  produced.  The  complexities  that  are  suggested  by  the  questions  I have  posed  for  both 
internal  and  external  validation  studies  can  also  lead  one  to  consider  the  possibility  that  ID  models  may  be  valid 
for  one  design  setting  and  not  for  another. 

Design  contexts  vary  not  only  in  terms  of  available  resources  and  facilities,  but  also  in  terms  of  the 
climate  and  emphases  imposed  by  factors  such  as  the  organization’s  mission  and  leadership  style.  They  also 
vary  in  terms  of  the  characteristics  of  the  learning  and  performance  environments  in  which  the  subsequent 
instruction  is  implemented.  The  many  aspects  of  context  that  impact  the  design  process  have  been  identified  and 
discussed  in  Tessmer  and  Richey  (1997).  Some  of  the  updated  ID  models  (e.g.  Dick,  Carey  & Carey,  2001) 
specifically  recognize  these  factors  and  include  procedures  for  dealing  with  them.  More  often  than  not, 
however,  designers  modify  the  generic  models  to  accommodate  their  unique  work  environments.  For  example, 
they  may  eliminate  or  curtail  the  analysis  phase  and  use  previously  collected  data  or  the  input  of  supervisors. 
Thus,  detailed  design  procedures  can  vary  depending  upon  context  even  when  the  same  model  is  being  used 
ostensibly. 

Designer  Expertise  Effects.  Design  models  are  also  typically  interpreted  differently  by  expert  designers 
and  novice  designers.  This  has  been  well  established  by  researchers  such  as  Rowland  (1993),  Perez  and  Emery 
(1995),  Saroyan  (1993)  and  others.  While  the  key  tasks  are  still  completed  by  experts,  they  tend  to  treat  all 
problems  as  ill-defined.  They  consider  a wide  variety  of  situational  factors  in  combination,  as  well  as  both 
instructional  and  non-instructional  solutions,  but  delay  making  design  decisions  as  long  as  possible.  Perez  and 
Emery  noted  that  experts  “ interpreted  the  design  problem  - novices  identified  the  problem”  (1995,  p.  92). 

Some  approaches  to  design  demand  more  experienced  designers  than  others.  This  is  true,  for  example, 
of  rapid  prototyping  procedures  (Jones  & Richey,  2000).  Expert  designers  basically  use  a general  ISD  model, 
but  the  design  process  tends  to  be  more  iterative,  and  the  sequencing  of  design  steps  varies  to  meet  the  demands 
of  an  individual  design  project.  Design  tasks  are  performed  concurrently.  This  procedure  is  common  in  larger 
organizations  where  projects  are  typically  completed  by  design  teams,  with  members  each  having  expertise  in  a 
unique  area  required  by  the  project  at  hand. 

Edmonds,  Branch  and  Mukherjee  (1994)  see  that  ID  models  themselves  are  often  oriented  toward 
either  expert  or  novice  designers.  Experts  use  intuitive  judgment  stemming  from  their  past  experiences  to 
provide  design  guidance.  The  design  experience  of  experts,  they  further  contend,  is  necessary  to  use  the  Layers 
of  Necessity  approach  to  design  (Tessmer  & Wedman,  1990)  as  well  as  the  rapid  prototyping  model  proposed 
by  Tripp  and  Bichelmeyer  (1990).  Another  explanation  of  expert  design  behavior  is  that  they  modify  standard 
models  to  meet  the  demands  of  a given  situation.  In  either  case,  designer  expertise  and  design  context  interact  to 
shape  the  design  task,  and  these  interactions  have  important  implications  for  the  systematic  validation  of  an  ID 
model. 


ID  Model  Validation  Procedures 

There  are  various  ways  of  conducting  both  internal  and  external  validations.  In  this  section  five 
different  validation  procedures  are  discussed.  These  alternative  approaches  can  be  viewed  as  types  of 
instructional  design  and  development  research  (see  Richey,  Klein  & Nelson,  2004).  These  validation  processes 
are  not  mutually  exclusive;  the  various  approaches  can  be  combined  to  reinforce  the  data  base  and,  in  turn,  to 
strengthen  the  conclusions.  Table  1 compares  each  of  these  validation  processes  in  terms  of  the  typical  types  of 
ID  models  addressed,  the  focus,  the  research  techniques  employed  and  when  the  validations  are  typically 
undertaken. 
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Table  1 A Comparison  of  Five  Approaches  to  ID  Model  Validation 


Validation  Process 

Types  of  ID 
Models 
Addressed 

Typical 

Focus 

Research  Techniques 
Employed 

Time  of 
Completion 

Internal  Validation: 
Expert  Review 

Conceptual 

Procedural 

Model 

Components; 
Model  Use 

Survey 

Delphi 

During  model 
development 

Usability 

Documentation 

Procedural 

Model  Use 

Case  Study 

During  model 
try-out  or  use 

Component 

Investigation 

Conceptual 

Procedural 

Model 

Components 

Survey 

Experimental  or 
Quasi-Exp. 

Path  Analysis 
LISREL  Analysis 

Prior  to  model 
development; 
During  model 
use 

External  Validation: 
Field  Evaluation 

Procedural 

Product 

Characteristics; 

Instructional 

Impact 

Case  Study 

Evaluation 

Survey 

During  model 
try-out  or  use 

Controlled  Testing 

Procedural 

Instructional 

Impact 

Experimental  or 
Quasi-Experimental 

During  model 
use 

Internal  Validation  Procedures 

Expert  Review.  One  of  the  most  commonly  used  approaches  to  internal  validation  is  expert  review. 
Expert  review  is  a process  whereby  ID  experts  critique  a given  model  in  terms  of  its  components,  overall 
structure  and  future  use.  It  is  the  most  expeditious  of  the  internal  validation  methods.  Essentially,  this  is  a 
cyclical  process  of  model  review  and  critiquing  based  upon  pre -specified  criteria,  and  subsequent  model 
revision  based  upon  the  data.  Often  Delphi  techniques  are  emp  loyed  as  a framework  for  achieving  consensus 
among  the  participants.  Participants  need  not  physically  meet;  instead,  data  are  typically  collected  via  mail,  e- 
mail,  telephone  or  web-based  instruments.  The  process  continues  until  there  is  a consensus  among  the  panel  of 
experts  as  to  the  completeness  and  the  utility  of  the  model.  Some  of  the  best  examples  of  expert  review 
validations  are  in  doctoral  dissertations.  For  example,  Sleezer  (1991)  developed  and  validated  a Performance 
Analysis  for  Training  (PAT)  Model  using  expert  review  methods.  She  used  experts  in  training  needs  assessment 
to  evaluate  the  content  and  face  validity  of  the  PAT  model. 

In  this  approach,  the  soundness  of  the  validation  is  dependent  to  a great  extent  upon  the  number  of 
reviewers  and  the  authority  of  the  reviewers.  This  validation  relies  upon  the  experiences  and  knowledge  of  the 
reviewers.  Often,  reviewers  represent  both  design  practitioners  and  design  theorists,  but  persons  are  also 
selected  so  that  a variety  of  theoretical  orientations  and  work  settings  are  represented.  It  is  increasingly 
important  to  include  experts  with  geographical  diversity,  taking  special  care  to  reflect  design  practice  in 
countries  other  than  the  United  States. 

One  can  typically  expect  expert  reviews  to  be  most  credible  with  respect  to  verifying  model 
components.  Unless  the  participating  experts  have  used  the  target  model  themselves  or  have  extensive  work 
experience  in  a given  environment,  their  predictions  of  model  use  may  be  open  to  question.  More  robust  data  on 
model  use  is  usually  gathered  from  documentation  of  the  actual  design  and  development  process  as  it  occurs. 

Usability  Documentation.  The  second  approach  to  internal  validation  involves  the  systematic 
documentation  of  designers  using  a particular  model.  This  involves  keeping  records  of  actual  implementation 
procedures,  time  taken,  resources  used,  problems  and  difficulties  encountered  using  the  model  and  resolutions 
of  these  problems.  It  involves  systematically  describing  the  background  and  abilities  of  those  involved  and  of 
the  work  environment.  While  in  the  past  this  process  has  typically  been  an  added  task  for  designers,  such 
documentation  is  more  common  today  as  organizations  strive  to  establish  quality  standards  and  gain  recognition 
of  their  standards  through  avenues  such  as  ISO  certification. 

The  integrity  of  usability  documentation  data  is  dependent  upon  its  authenticity  and  objectivity.  Care 
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must  be  taken  to  insure  objectivity  through  consistent,  systematic  data  collection  techniques  and  the  collection 
of  corroborating  data.  Often  structured  logs  and  diaries  completed  by  several  project  participants  according  to  a 
regularly  established  schedule  create  a structure  that  facilitates  the  generation  of  reliable  data.  Recall  data 
should  be  avoided  when  possible. 

It  is  possible  that  usability  documentation  research  is  being  done  within  large  corporations  to  examine 
their  own  model  use.  If  so,  it  is  unlikely  that  these  studies  would  be  published  and  available  to  the  larger  ID 
community.  The  examples  of  usability  documentation  vary  considerably.  For  example,  Forsyth’s  (1998) 
usability  data  describes  the  specific  steps  completed  when  following  her  model  for  designing  community-based 
train -the-trainer  programs,  the  time  allocated  to  each  phase  of  the  model  and  the  lessons  learned  throughout. 

The  researcher  and  the  designer  in  this  case  were  one  in  the  same.  Data  were  obtained  from  logs  and  designer 
reflection. 

Both  expert  review  and  user  documentation  validation  schemes  depend  primarily  upon  reaction  data.  A 
key  difference  between  them  is  that  the  latter  demands  actual  use  of  the  model,  while  expert  review  data 
requires  reflection  and  analysis.  Component  investigation,  on  the  other  hand,  typically  involves  research  with  a 
fairly  rigorous  statistical  verification  of  the  factors  addressed  in  a given  design  model. 

Component  Investigation.  ID  models  have  many  parts.  In  general,  procedural  models  consist  of 
separate  steps,  and  conceptual  models  consist  of  factors  critical  to  the  instructional  design  process.  Each  of 
these  elements  can  be  initially  identified  or  confirmed  through  research.  This  is  the  essence  of  component 
investigation. 

The  research  directed  toward  procedural  model  validation  seeks  to  provide  evidence  of  the 
effectiveness  of  the  various  steps  in  the  process.  For  example,  very  early  in  the  use  of  ISD  procedures,  Kibler, 
Cegala,  Barker  and  Miles  (1974)  sought  to  establish  an  empirical  base  for  the  use  of  behavioral  objectives.  The 
research  directed  toward  conceptual  model  validation,  on  the  other  hand,  typically  seeks  to  identify  variables 
that  predict  key  outcomes  of  instruction  - either  knowledge  acquisition,  attitude  change  or  performance  change. 
These  predictive  variables  then  become  factors  that  should  be  addressed  in  the  design  of  instruction.  For 
example,  various  studies  have  validated  the  ARCS  Model  of  Motivation  Design  by  studying  the  impact  of  the 
various  model  components  (i.e.  attention,  relevance,  confidence  and  satisfaction)  on  achievement  (Brolin, 
Milheim  & Viechnicki,  1993-94;  Means,  1997;  Small  & Gluck,  1994). 

Other  studies  have  tested  an  array  of  variables  that  are  hypothesized  to  predict  successful  learning.  For 
example,  Quinones,  Sego,  Ford  and  Smith  (1995/1996)  in  their  investigation  of  transfer  of  training  used 
LISREL  analysis  techniques  to  support  a model  of  factors  that  predict  the  opportunity  to  perform  in  a work 
environment  after  training.  Noe  and  Schmitt  (1986)  used  path  analysis  techniques  to  develop  models  of  training 
effectiveness.  The  strength  of  these  findings  as  validation  tools  is  naturally  dependent  upon  the  integrity  of  the 
foundational  research. 

These  first  three  approaches  to  ID  model  validation  are  internal,  speaking  only  to  the  worth  of  the 
model  itself  without  examining  the  results  of  using  the  model  on  learners  or  on  the  organizations  in  which  they 
are  used.  Approaches  that  address  these  latter  issues  are  external  in  nature. 

External  Validation  Procedures 

Field  Evaluation.  Field  evaluation  is  the  most  commonly  used  external  validation  process.  As  with 
usability  documentation,  it  involves  actually  using  the  model  to  produce  instruction.  However,  here  the 
instructional  product  is  also  implemented  in  the  setting  for  which  it  was  designed.  Data  is  collected  to  facilitate 
a study  of  the  nature  of  the  resulting  product  and  the  impact  of  the  instruction  on  learners  and  organizations. 

The  impact  of  the  entire  process  on  clients  is  also  documented  in  some  situations. 

Sullivan,  Ice  and  Niedermeyer  (2000)  systematically  field  tested  a comprehensive  K-12  energy 
education  curriculum  that  tested  a long-term  instructional  development  and  implementation  project.  The  project 
has  been  on-going  for  20  years.  Conclusions  were  drawn  that  could  be  generalized  to  other  long-term  projects. 
McKenney  (2002)  validated  her  design  and  development  model  through  an  extensive  field  evaluation.  She 
studied  the  development  of  a computer  program  to  support  curriculum  materials  development  in  the  context  of 
secondary  science  and  mathematics  education  in  southern  Africa.  Superior  field  evaluations  draw  upon  the 
methodologies  of  any  product  or  program  evaluation  effort.  However,  when  used  for  ID  model  validation  the 
results  need  to  be  examined  in  terms  of  their  implications  for  confirming  or  altering  the  basic  design  model  that 
guided  the  project. 

Controlled  Testing.  Design  models  can  also  be  validated  by  establishing  experiments  that  isolate  the 
effects  of  the  given  ID  model  as  compared  to  the  use  of  another  model  or  approach.  This  is  the  object  of 
controlled  testing  validation.  This  type  of  research  provides  data  that  supports  the  validity  of  a given  procedural 
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model  under  controlled  conditions. 

There  are  examples  of  controlled  model  testing  available.  Tracey  (2002)  is  one  example.  She  compared 
the  use  of  the  Dick  and  Carey  model  with  an  ISD  model  enhanced  with  a consideration  of  multiple 
intelligences.  The  experiment  was  controlled  in  terms  of  task,  time,  and  designer  expertise.  In  the  Tracey  study 
there  were  two  design  teams,  each  with  two  novice  designers.  Each  team  worked  with  a different  model.  Both 
groups  designed  a two -hour  instructor-led  workshop.  The  resulting  programs  were  actually  implemented  and 
evaluated.  The  verification  of  model  effectiveness  was  based  upon  measures  of  learning  and  participant 
reactions  to  the  instruction.  Research,  such  as  this,  is  more  likely  to  be  undertaken  by  academics  than 
practitioners,  but  when  it  can  take  place  in  natural  work  environments  the  results  are  apt  to  be  seen  as  more 
trustworthy  among  practicing  designers. 

Comprehensive  Model  Validation 

Given  the  contextual  nature  of  ID  model  use,  there  is  the  question  whether  model  validation  findings 
can  be  generalized  to  other  settings.  This  concern  speaks  to  the  need  for  comprehensive  validation  efforts.  All 
of  the  examples  discussed  have  been  situation-specific.  This  reflects  the  nature  of  the  vast  majority  of  validation 
efforts.  Comprehensive  validations,  on  the  other  hand,  would  examine  ID  model  use  under  a variety  of 
conditions.  To  meet  these  goals,  the  validation  research  requires  systematic  replication.  Systematic  replication 
of  ID  model  validation  research  would  allow  the  field  to  determine  the  impact  of  factors  such  as: 

• alternative  settings; 

• alternative  types  of  learners; 

• designer  expertise; 

• alternative  content  areas;  and 

• a variety  of  delivery  strategies 

It  seems  to  make  the  most  sense  for  comprehensive  research  to  be  conducted  during  model  use  (e.g.,  usability 
documentation  studies,  field  evaluations,  and  controlled  testing)  rather  than  model  development. 

When  a large  number  of  validation  studies  appear  in  the  ID  literature  it  will  be  possible  to  employ 
meta-analytic  or  other  integrative  research  procedures  using  these  findings  to  summarize  the  findings  pertaining 
to  the  effectiveness  of  a particular  ID  model  that  has  been  employed  in  a variety  of  situations.  Replication  of 
model  effectiveness  under  a variety  of  conditions  and  integration  of  the  various  findings  would  not  only 
increase  the  credibility  of  that  particular  model,  but  would  provide  data  to  support  the  assumption  that  ISD 
processes  are  generic. 


Conclusions 

The  underlying  theme  of  this  paper  is  that  as  a field  we  should  be  validating  our  many  ID  models  as 
well  as  developing  them,  and  that  validation  should  become  a natural  part  of  the  model  development  process. 
This  message  stems  from  a belief  that  instructional  design  itself  is  a science,  but  one  that  is  practiced  by 
blending  creative  and  rule -bound  activities.  By  viewing  design  itself  as  a science  rather  than  an  extension  of 
science,  it  follows  that  its  overarching  theories  and  models  should  be  grounded  not  only  in  research  on 
instruction,  but  in  research  on  instructional  design  and  development. 
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Abstract 

The  purpose  of  this  study  was  twofold:  a)  to  test  the  hypothesis  that  participation  in  a course  providing 
theory  and  practice  in  the  classroom  use  of  technology  engenders  such  use  and  results  in  gained  competence  in 
the  use  of  this  technology;  and  b ) to  test  the  hypothesis  that  functional  relations  exist  among  the  availability  of 
technology  in  the  classroom,  the  use  of  such  classroom  technology,  and  competence  in  its  use.  Fifty-five 
graduate  students  participated  in  a course  in  the  use  of  technology  in  the  classroom.  Statistically  significant 
differences  emerged  between  pre-tests  and  post-tests  on  a)  use  of  e-mail,  b)  Web  site  use,  c)  automated  library 
resources,  d)  computer  software,  e)  Internet  virtual  field  trips,  f)  video  cameras,  g)  PowerPoint,  and  h) 
Blackboard — pointing  to  the  effectiveness  of  the  course  for  the  purpose.  Further,  as  hypothesized,  competence 
in  the  use  of  technology  in  the  classroom  proved  to  be  a direct  function  of  the  degree  of  use  of  this  technology. 


Introduction 

Teachers  are  expected  to  use  technology  in  their  classrooms.  However,  teachers  who  have  been  in  the 
field  for  more  than  10  years  often  do  not  have  the  necessary  expertise  to  use  technology  in  the  classroom.  Their 
education  coursework  may  not  have  included  technology  training,  and  these  teachers  may  not  have  obtained  the 
necessary  technology  skills  on  their  own.  Further,  many  schools  have  been  lax  in  offering  their  faculty  members 
technology  training  with  appropriate  follow-up  and  support. 

Teachers  who  have  had  “training”  often  report  that  it  consisted  of  a single  in-service  session  with  no 
follow-up  or  support.  Teachers  who  are  in  school  to  earn  master’s  degrees  are  sometimes  required  to  take  a 
course  to  help  them  integrate  technology  into  their  own  classrooms.  The  course  includes  theory  as  well  as 
opportunities  to  practice  newly  learned  skills  to  achieve  competence  with  technology.  Both  the  course  instructor 
and  class  members  provide  ongoing  support  for  novice  technology  users.  Questions  arise  whether  these  courses 
really  enable  teachers  to  gain  competence  in  the  use  of  technology  so  that  they  can  integrate  their  newly  learned 
technology  expertise  into  their  classrooms. 


Literature  Review 

There  is  wide  variance  in  the  nature  of  the  training  to  enable  teachers  to  use  technology  tools  in  their 
classrooms.  Researchers  report  that  teacher-training  programs  generally  fail  to  provide  future  teachers  with  the 
experiences  necessary  to  prepare  them  to  effectively  use  technology  in  their  classrooms  (Baylor  & Ritchie, 

2002;  Clouse  & Alexander,  1998;  Ertmer,  Conklin,  & Lewandowski,  2001;  Russell,  Bebell,  O’Dwyer,  & 
O’Connor,  2003;  Wiencke,  2002).  Although  newer  teachers  are  usually  comfortable  working  with  technology, 
they  have  not  been  exposed  to  applications  of  classroom  technology.  These  newer  teachers  have  more  recently 
completed  teacher  education  programs,  many  of  which  focus  on  how  to  use  technology  rather  than  on  how  to 
teach  with  technology  and  integrate  it  into  everyday  teaching.  Models  of  teaching  based  on  their  own 
experiences  as  students  do  not  include  the  integration  of  technology  into  instruction.  Russell,  Bebell,  O’Dwyer, 
and  O’Connor  further  posit  that  although  teachers  use  technology  outside  the  classroom,  especially  for 
preparation  and  professional  communication  via  e-mail,  they  infrequently  use  technology  in  the  classroom. 
Ertmer,  Conklin,  and  Lewandowski  (2001)  comment  that  knowing  how  to  use  word -processing,  e-mail,  and  the 
Internet  does  not  mean  facilitation  of  these  skills  will  occur  in  classroom  instruction.  Once  teachers  are  in  the 
classroom,  opportunities  for  learning  to  use  technology  are  rare  since  most  in-service  training  programs  lack  a 
model  for  integration  (Clouse,  & Alexander,  1998)  and  many  schools  have  not  yet  incorporated  technology  into 
regular  instruction  (Russell,  Bebell,  O’Dwyer,  & O’Connor,  2003).  Gooler,  Kautzer,  and  Knuth,  (2000)  explain 
that  the  teacher  plays  a key  role  in  determining  not  only  how  but  how  well  technologies  are  used  in  classrooms, 
and  thus  the  extent  to  which  technologies  improve  student  performance.  Khamis  (1987)  reports  that  merely 
placing  a computer  into  a classroom  of  untrained  teachers  is  ineffective  since  untrained  teachers  are  likely  to 
use  computers  for  daily  trivial  things.  These  inconsistencies  point  to  the  need  for  developing  teachers’ 
classroom  technology  competence. 
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Teachers  need  both  the  time  and  the  opportunity  to  gain  competence  in  instructional  technology 
(Harris,  2000).  Some  researchers  report  that  teachers  progress  through  stages  as  they  develop  technology 
integration  competence.  At  each  stage  teachers  need  particular  support  and  professional  development  (Gooler, 
Kautzer,  & Knuth  , 2000;  Harris,  2000).  A compilation  of  research  suggests  that  effective  professional 
development  in  learning  to  use  technology  to  teach  should  have  relevance  for  the  teacher  and  include  modeling, 
hands-on  practice,  continuing  support,  collaboration,  and  easy  access  to  the  technology. 

Modeling.  As  mentioned  earlier,  many  teachers  do  not  have  good  models  for  integrating  technology 
into  their  teaching.  Baylor  and  Ritchie  (2002)  note  that  teachers  tend  to  teach  the  way  they  were  taught. 
Providing  appropriate  models  so  that  teachers  can  observe  and  then  practice  themselves  is  useful  for  many 
teaching  applications  (Dunne  & Harvard,  1992).  Clouse  and  Alexander  (1998)  claim  that  the  best  training  is 
through  observation  and  collaboration  with  full  time  teachers  who  use  technology  effectively  in  their 
classrooms. 

Mager  (1992)  suggests  that  modeling  by  peers  is  a good  training  strategy  to  help  self-efficacy  (a 
personal  judgment  about  one’s  ability  to  carry  out  a particular  course  of  action  or  do  a specific  thing.)  Although 
the  present  study  focuses  on  competence,  it  should  be  noted  that  self-efficacy/confidence  is  an  important 
consideration  for  teachers  in  their  decision  to  use  technology  in  their  classrooms. 

Hands-on  practice.  Before  teachers  can  infuse  technology  into  the  curriculum,  they  need  to  have 
appropriate  skills,  knowledge  and  attitude  (Baylor  & Ritchie,  2002;  Gooler,  Kautzer,  & Knuth,  2000).  Clouse 
and  Alexander  (1998)  suggest  that  the  most  effective  training  programs  must  provide  practical  hands-on 
experiences  and  meaningful  activities  that  are  appropriate  for  an  individual’s  level  of  expertise.  Teachers  need 
time  to  reflect  on  new  learning  and  integrate  this  new  knowledge  into  practice  through  experimentation  and  then 
reflect  on  these  outcomes  further  so  that  appropriate  adjustments  can  be  made  (Gooler,  Kautzer,  & Knuth, 

2000).  Khamis  (1987)  agrees  that  teachers  need  time  to  practice  to  improve  their  competence  and  further 
suggests  a team  strategy  of  requiring  teachers  to  participate  in  student  activities  led  by  a more  experienced  team 
member. 

Continuing  support.  Continuing  support  is  an  important  ingredient  if  teachers  are  to  use  technology  in 
the  classroom  (Gooler,  Kautzer,  & Knuth,  2000).  Introductory  teacher  training  is  unlikely  to  guarantee 
continued  use  of  technologies.  Support  is  needed  to  help  teachers  infuse  technology  into  the  curriculum  as  well 
as  to  provide  technical  expertise  to  insure  that  the  equipment  is  functioning  properly. 

Collaboration.  Well  designed  training  programs  provide  opportunities  for  participants  to  interact  and 
collaborate  so  that  they  can  learn  together  and  from  each  other  (Gooler,  Kautzer,  & Knuth,  2000).  A collegial 
and  collaborative  culture  in  which  colleagues  can  exchange  knowledge  and  ideas  and  provide  constructive 
feedback  and  encouragement  to  their  peers  fosters  the  growth  of  (and  is  cultivated  in)  a learning  community. 
Teachers  should  be  in  a supportive  environment  when  trying  something  new  (Harris,  2000). 

Ease  of  access.  Teachers,  administrators,  and  students  who  have  easy  access  to  technology  are  more 
likely  to  take  the  time  to  practice  with  it  to  improve  their  skills  (Khamis,  1987).  Harris  (2000)  reports  on 
exemplary  uses  of  technology  in  several  school  projects  in  southeastern  states.  In  one  program  participating 
teachers  received  laptops;  in  another  project  there  were  two  computers  in  each  classroom,  and  in  yet  another 
project  there  was  a two  to  one  ratio  of  students  to  computers.  Harris  reasons  that  easy  and  regular  access  to 
computers  is  necessary  if  teachers  are  to  plan  lessons  requiring  children  to  use  the  Internet  or  to  prepare  reports 
using  presentation  software. 


Purpose 

The  purpose  of  this  study  was  twofold:  a)  to  test  the  hypothesis  that  participation  in  a course  providing 
theory  and  practice  in  the  classroom  use  of  technology  engenders  such  use  and  results  in  gained  competence  in 
the  use  of  this  technology;  and  b)  to  test  the  hypothesis  that  functional  relations  exist  among  the  availability  of 
technology  in  the  classroom,  the  use  of  such  classroom  technology,  and  competence  in  its  use. 

The  model  in  Figure  1 was  used  to  guide  the  research  effort.  In  this  model,  we  hypothesized  that 
Availability  of  Technology  (AT)  would  engender  the  Use  of  Technology  (UT),  that  UT  would  engender  the  Use 
of  Internet  Web  sites  (UW),  and  that  these  conditions  would  engender  Competence  in  the  Use  of  technology 
(CU). 


Method 

The  study  was  conducted  during  the  Fall  2002  and  Spring  2003  semesters  in  three  graduate  educational 
technology  classes.  One  instructor  taught  all  the  classes  . 
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Sample 

Fifty-five  graduate  students  (master’s  students  in  education)  participated  in  the  study.  Most  of  these 
graduate  students  were  employed  as  teachers  in  an  urban  school  district.  Participant  technology  expertise  ranged 
from  almost  no  technology  background  to  those  who  were  facilitating  students’  classroom  use  of  technology. 
The  availability  of  technological  resources  for  teachers  in  their  schools  ranged  from  no  technology  resources  to 
several  computers  in  their  classrooms. 

Procedure 

A variety  of  activities  were  designed  to  give  participants  hands-on  experiences  with  the  technology,  as 
recommended  in  the  research.  Participants  found  information  on  Web  sites  and  navigated  electronic  library 
resources.  They  learned  to  send  file  attachments  and  became  proficient  with  e-mail.  They  researched  a topic  and 
used  PowerPoint  to  develop  a presentation.  Participants  prepared  and  shared  virtual  field  trip  lessons  using 
topics  suitable  for  their  own  students. 

Collaboration  was  encouraged,  recognizing  a preference  to  work  with  a partner  when  learning 
something  new  (Rosenfeld,  1992).  PowerPoint  assignments  were  completed  in  small  groups  and  presented  to 
the  class.  It  was  anticipated  that  being  a part  of  a group  would  be  less  threatening  than  developing  and  making  a 
solo  presentation,  particularly  since  the  PowerPoint  program  was  new  to  many  participants.  The  virtual  field 
trip  (VFT)  lessons  were  presented  individually,  but  participants  could  opt  to  collaborate  on  the  development  of 
the  plans.  The  VFT  lessons  were  presented  to  small  groups  so  that  this  would  be  a less  threatening  environment 
for  those  who  had  newly  learned  about  this  technology-driven  activity.  The  presentations  gave  participants  the 
opportunity  to  try  out  the  technology  with  an  audience  (practice),  and  also  provided  participants  with  several 
models  to  emulate,  as  recommended  in  the  research. 

The  use  of  Blackboard  (a  distance  learning  management  program)  allowed  for  continued  discussion 
outside  class  on  the  discussion  board  or  through  small  group  discussion  forums.  Weekly  announcements  were 
publicized  to  relay  information  about  the  class  prior  to  meeting.  Grades  and  course  documents  (e.g.,  syllabus 
and  class  handouts)  were  posted.  Blackboard  gave  participants  another  opportunity  to  build  their  technology 
competence,  allowing  the  classroom  community  to  meet  asynchronously. 

Instrumentation 

The  Survey  of  Technology  Use  Questionnaire  (STUQ)  was  administered  at  the  beginning  of  each 
semester  (STUQ  1),  and  again  at  the  end  of  each  semester  (STUQ  2).  Participants  were  queried  regarding  their 
perceived  competence  in  the  use  of  e-mail,  Web  site  use,  automated  library  resources,  computer  software, 
Internet  virtual  field  trips  (VFTs),  video  cameras,  PowerPoint,  and  Blackboard/WebCT.  These  activities  were 
grouped  into  two  categories,  Technology  Use  (automated  library  resources,  computer  software,  video  cameras, 
PowerPoint)  and  Internet  and  Web  Site  Use  (Blackboard,  e-mail,  VFTs).  The  information  gleaned  with  STUQ  1 
and  STUQ  2 was  deemed  as  indicative  of  the  participants’  use  of  and  perceived  competence  with  the  various 
technologies  before  and  after  participation  in  the  study. 

Additional  data  was  collected  through  a course  feedback  survey.  Questions  asked  about  participants’ 
classroom  use  of  technology  at  the  end  of  the  course,  for  example,  whether  they  had  tried  a virtual  field  trip 
(VFT)  prior  to  the  class  and  whether  they  had  tried  a VFT  since  it  had  been  discussed  in  class. 

Method  of  Data  Analysis 

To  test  the  first  hypothesis  posed  for  investigation,  1-tailed  paired  comparison  f-tests  were  performed 
to  ascertain  gains  in  the  use  of  technology  in  the  classroom  and  competence  in  the  use  of  this  technology.  To 
address  the  second  hypothesis,  path  analysis  was  performed  to  test  the  model  as  shown  in  Figure  1. 

Results 

The  f-test  outcomes  appear  in  Table  1.  As  shown  in  this  table,  a statistically  significant  gain  emerged 
for  competence  in  the  use  of  technology  from  pre-  to  post-test.  (CU)  from  STUQ  1 (M  = 11.85,  SD  = 5.34)  to 
STUQ  2 (M  = 19.10,  SD  = 3.34):  t = -11.06,  df  = 47,  p = .00) — suggesting  that  the  interventions  were 
successful  in  raising  the  participants’  competence  in  using  technology  in  the  classroom. 
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Table  1.  t-Test  Outcomes 


M 


SD 


P 


Pre-Test  11.85  5.34  -11.06  .00 

Post-Test  19.10  3.34 

Figure  1 shows  the  path  analysis  outcomes.  As  hypothesized,  competence  in  the  use  of  technology  in 
the  classroom  proved  to  be  a direct  function  of  the  degree  of  use  of  this  technology,  most  directly  on  the  use  of 
Internet  Web  sites  ((3  = .56,  p < .05);  the  degree  to  which  teachers  use  the  Internet  proved  to  be  dependent  on 
both  the  availability  of  computer  technology  in  the  schools  ((3  = .58,  p < .05),  and  the  general  use  of  this 
technology  ((3  = .42,  p < .05);  and  the  use  of  this  technology  proved  to  be  a function  of  the  degree  to  which  it  is 
available  to  teachers  in  the  schools  ((3  = .77,  p < .05). 

Figure  1.  Hypothetical  Model  and  Path  Analysis  Outcomes.  The  coefficients  for  the  heavy-lined 
linkages,  in  the  form  of  standardized  regression  weights,  are  statistically  significant  beyond  the  .05 
level. 


'Automated  library  resources,  computer  software,  video  cameras,  PowerPoint 
2BlackBoard  (distance  learning  software),  e-mail,  virtual  field  trips. 


Additional  information  from  the  course  feedback  survey  indicates  that  by  the  end  of  the  course  some 
participants  were  infusing  technology  into  their  own  classrooms  while  other  participants  expressed  an  interest  in 
doing  so,  but  could  not  because  of  a variety  of  problems  barring  classroom  technology  use. 

Discussion 

The  interventions  used  in  this  study  took  into  account  suggestions  from  past  research.  Professional 
development  should  be  relevant  to  the  teachers  and  incorporate  modeling,  hands-on  practice,  continuing 
support,  collaboration,  and  easy  access  to  the  technology.  In  most  instances,  participants  chose  topics  for  their 
projects  and  assignments  resulting  in  a course  that  had  relevance  for  the  participants.  For  example,  the  VFT 
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lesson  plans  were  developed  for  use  in  participant  teachers’  own  classrooms.  One  participant  commented,  “You 
tried  to  make  this  class  useful  for  our  professional  lives.  You  did  not  have  us  working  in  hypotheticals  which  I 
really  appreciated”  (Participant’s  response  to  the  course  feedback  survey.  Spring,  2003).  Another  observed, 

“We  were  exposed  to  many  useful  and  interesting  activities  that  would  all  be  wonderful  to  use  with  kids” 
(Participant’s  response  to  the  course  feedback  survey,  Spring,  2003).  A third  participant  wrote. 

We  were  involved  doing  projects  where  we  could  see  practical  use  in  the  classroom.  In  the  other 
education  classes  I have  taken,  they  gave  us  a lot  of  theory  about  instruction  and  management,  but 
these  were  things  that  we  could  use  right  away.  I guess  you  applied  the  theories  with  us  that  we  are 
being  told  to  use.  We  are  being  taught  to  deliver  the  instruction  in  a way  that  makes  it  relevant  to  the 
students.  This  was  very  relevant  to  our  situations.  You  also  made  sure  that  the  course  did  not  just  get 
wrapped  up  in  the  technology.  There  were  always  connections  to  the  educational  reasons  for  using  the 
technology  (Participant’s  response  to  the  course  feedback  survey,  Spring,  2003). 

Modeling  and  hands-on  practice  were  provided  through  in-class  presentations  - participants  had  hands- 
on  experiences  developing  PowerPoint  presentations  and  they  modeled  technology  use  for  each  other.  A 
participant  remarked,  “I  liked  both  the  PowerPoint  and  the  VFT.  Both  presentations  were  educative  and 
informative.  In  addition,  these  assignments  enabled  us  to  learn  from  each  other”  (Participant’s  response  to  the 
course  feedback  survey,  Spring,  2003).  Another  echoed  that  she  “loved  the  opportunities  that  we  had  as  a class 
to  learn  from  each  other”  (Participant’s  response  to  the  course  feedback  survey,  Spring,  2003). 

During  the  semester  there  was  continuing  support  from  the  instructor  as  well  as  from  classmates.  A 
collaborative  environment  was  encouraged  with  many  opportunities  for  participants  to  work  together  in  class 
and  asynchronously  through  Blackboard.  A participant  wrote,  “I  enjoyed  all  the  group  activities.  I love 
exchanging  thoughts  and  ideas  with  others”  (Participant’s  response  to  the  course  feedback  survey.  Fall,  2002). 
Another  wrote,  “I  have  truly  enjoyed  working  together  with  the  other  students.  With  their  different  background 
and  knowledge  I have  gained  great  knowledge  for  myself  and  my  students.  Every  time  I had  a chance  to  work 
with  them,  I felt  like  I came  out  of  that  experience  even  stronger  and  better”  (Participant’s  response  to  the 
course  feedback  survey,  Fall,  2002). 

The  class  met  in  a computer  lab,  providing  easy  access  to  the  technology,  but  outside  of  the  class 
participants  worked  in  varied  environments.  All  participants  reported  having  a computer  at  home  and  all  had 
Internet  access  either  at  home  or  at  school.  Of  the  53  respondents  to  the  end  of  semester  Survey  of  Technology 
Use  Questionnaire  (STUQ  2),  eight  reported  that  they  had  no  computer  access  at  school;  three  additional 
respondents  had  no  Internet  access  at  school.  This  means  that  over  20%  of  the  participants  in  this  study  could 
not  facilitate  on-line  activities  in  their  classes. 

It  should  be  noted  that  available  technology  is  not  always  in  good  working  order.  One  participant 
reported  that  of  four  computers  in  his  classroom  only  one  worked.  Teachers  often  recount  that  the  school  does 
not  provide  quick  technical  support  to  fix  computers  when  they  malfunction.  This  may  cause  teachers  to  avoid 
the  use  of  technology  altogether.  When  teachers  report  on  the  number  of  computers  that  are  in  their  classrooms, 
they  also  should  report  on  the  status  of  these  computers  - are  they  in  working  condition?  How  old  are  they?  Do 
they  have  the  memory  and  speed  to  support  graphical  and  audio  downloads?  Technology  that  does  not  match 
users’  needs  will  not  be  used. 

Responses  on  the  course  feedback  survey  indicate  that  1 1 participants  tried  a VFT  with  their  students 
since  that  topic  was  discussed  in  class,  18  planned  to  do  so  in  the  future,  9 reported  that  the  VFT  was  not 
available  or  accessible,  and  1 1 responded  that  they  would  like  to  but  could  not.  There  were  8 affirmative 
responses  to  the  question  on  using  PowerPoint  with  students  since  the  topic  was  discussed  in  class,  12  said  that 
they  plan  to  use  PowerPoint  in  the  future,  andlO  reported  that  they  would  like  to  use  PowerPoint  but  could  not. 
Twelve  respondents  said  that  they  had  tried  other  technology  activities  with  students,  including  videotaping  and 
showing  videotapes,  using  software,  spreadsheets,  a digital  camera,  Web  sites  and  other  Internet  resources. 

Information  from  the  course  feedback  survey  indicates  that  by  the  end  of  the  course  some  participants 
were  infusing  technology  into  their  own  classrooms  while  other  participants  expressed  an  interest  in  doing  so, 
but  could  not  because  of  a variety  of  problems  barring  classroom  technology  use.  In  addition  to  inaccessibility, 
lack  of  Internet  access,  and  malfunctioning  computers,  some  participants  reported  that  they  were  working  on 
city-wide  testing,  were  not  currently  teaching,  had  scheduling  problems,  lacked  software,  or  had  very  young 
students. 

There  may  be  other  factors  that  prevent  teachers  from  using  technology  in  the  classroom.  For  example, 
even  when  teachers  are  competent  in  technology  use,  if  they  do  not  have  self-efficacy  or  confidence,  they  may 
not  opt  to  try  to  use  technology  tools  in  the  classroom.  Several  participants  mentioned  in  the  course  feedback 
that  they  felt  more  confident  and/or  comfortable  with  technology  use:  “I  have  not  been  in  school  for  13  years  so 


747 


I was  a little  intimidated  about  having  to  go  back  but  you  made  me  feel  very  comfortable  and  confident”  and  ‘‘I 
learned  a lot  of  new  things  for  myself  and  to  incorporate  in  my  classroom.  It  has  made  me  more  comfortable 
with  technology”  (Fall  2002).  Another  participant  commented,  ‘‘This  course  helped  me  be  a confident  user  of 
the  computer”  (Spring  2003).  The  present  study  only  looked  at  perceived  competence  with  technology.  A future 
study  might  also  examine  self-efficacy/confidence  and  comfort  issues. 

Conclusion 

The  statistical  data  support  the  first  hypothesis,  that  participation  in  a course  providing  theory  and 
practice  in  the  classroom  use  of  technology  engenders  such  use  and  results  in  gained  competence  in  the  use  of 
this  technology.  The  data  also  support  the  second  hypothesis,  that  functional  relations  exist  among  the 
availability  of  technology  in  the  classroom,  the  use  of  such  classroom  technology,  and  competence  in  its  use. 

The  results  of  this  study  strongly  suggest  that  when  an  educational  technology  course  is  relevant  to  students’ 
needs  and  provides  students  with  modeling,  hands-on  practice,  and  continuing  support  in  a collaborative 
environment,  it  can  lead  to  competence  and  future  classroom  technology  use.  This  study  also  points  to  the  need 
for  states  to  mandate  that  a similar  course,  in-service,  or  training  be  required  of  all  teachers  who  are  expected  to 
use  technology  tools  in  the  classroom.  Further,  principals  and  other  administrators  should  support  their  teachers’ 
efforts  to  infuse  technology  in  their  classrooms,  thus  helping  their  students  to  better  prepare  for  the  future. 
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Problem  Statement 

Current  theories  of  learning  have  emphasized  the  value  of  dialogue  for  student  engagement  and 
achievement  (Cazden,  2001;  Bruffee,  1992).  Research  has  also  shown  that  the  nature  of  classroom  discourse 
depends  greatly  on  the  teacher  (Anderson,  Rourke,  Garrison,  & Archer,  W.,  2001).  These  issues  are  relatively 
well  understood  in  face-to-face  classrooms.  However,  the  advent  of  online  learning  has  raised  more  questions 
about  student  interaction  and  the  role  of  teachers  in  such  contexts.  We  need  to  develop  a better  understanding  of 
how  teachers  can  provide  effective  online  mentoring  and  scaffolding  to  facilitate  student  engagement  with  each 
other  and  with  their  subject  matter  (Bonk,  2003). 

Perceptions  of  online  teachers’  roles  in  distance  education  remain  quite  varied  and  controversial 
(Lobel,  Neubauer,  & Swedburg,  2002).  Although  online  instruction  literature  increasingly  emphasizes  the 
importance  of  moderation  and  leadership  (Anderson,  et  ah,  2001;  Feenberg,  1989),  it  remains  unclear  how 
online  moderating  are  related  to  student  engagement  and  interaction.  The  purpose  of  this  research  is  to  develop 
a better  understanding  of  the  relationship  between  moderator  behavior  and  student  engagement  in  synchronous 
computer  conferencing  learning  environments. 

Theoretical  Perspectives 

Current  interest  in  collaborative  learning — both  in  face-to-face  and  computer- mediated  classrooms — is 
grounded  in  socio -cultural  and  socio -constructive  theories  (Duffy  & Cunningham,  1996;  Vygotsky,  1934, 

1978).  Learning  is  seen  as  a process  of  negotiating  community  membership  through  various  social  interactions 
(Wenger,  1998)  with  peers,  experts,  and  teachers  (Kaye,  1992).  Proponents  of  computer  conferencing  have 
often  argued  that  such  social-constructivist  perspectives  may  be  particularly  amenable  to  this  new  medium 
(Bonk  & King,  1998;  Bruce  & Levin,  1997).  They  argue  that  computer  conferencing  may  help  students 
maximize  both  their  own  and  peers’  learning  through  the  use  of  collaborative  activities  and  discussions. 

The  teacher  plays  an  important  role  in  online  discussions.  While  individual  learning  can  occur  through 
independent  or  self-directed  study,  it  is  only  through  active  intervention  of  a teacher  or  moderator  that 
collaborative  computer  conferencing  becomes  a useful  instructional  and  learning  resource  (Garrison,  Anderson, 
& Archer,  2001).  Though  the  literature  recommends  (e.g.  Garrison  & Anderson,  2003;  Salmon,  2000)  extensive 
online  moderating  and  guidelines,  few  experimental  studies  evaluate,  much  less  certify,  moderating  processes  or 
validate  the  optimal  level  or  scope  of  online  moderating. 

Research  Questions 

As  we  have  argued  above,  the  relationship  between  moderating  level  and  student  engagement  is 
complicated.  Thus  the  major  task  of  this  study  is  to  investigate  the  relationship  between  teacher  moderating 
levels  and  student  engagement.  Taking  a mixed  research  method  approach,  both  quantitative  and  qualitative 
questions  are  asked: 

1.  Quantitatively,  how  are  teacher  moderating  levels  associated  with  each  of  the  three 

student  engagement  variables?  Is  the  effect  of  moderating  levels  on  one  student  engagement 

variable  higher  than  on  another  student  engagement  variable? 

2.  Qualitatively,  what  does  the  process  of  the  collaborative  meaning  construction  look 
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like?  What  is  the  transactional  nature  of  the  relationship  between  teacher  moderating 
levels  and  student  engagement? 

As  the  first  step  of  a large  scale  study,  our  answers  to  the  research  questions  are  based  on  a 
preliminary  analysis  of  a small  portion  of  the  large  data  set.  Additional  analyses  are  underway  and  will 
be  presented. 


Research  Design 

The  current  research  applies  a mixed  method  approach — a combination  of  qualitative  and  quantitative 
approaches.  Its  quantitative  character  is  evident  in  the  process  of  converting  communication  content  into 
discrete  units  and  calculating  the  frequency  of  occurrence  of  each  unit.  It  is  quantitative  also  in  that  it  extends 
the  descriptive  results  of  content  analysis  to  inferential  hypothesis  testing  (Borg  & Gall,  1989;  Rourke, 
Anderson,  Gairison,  Archer,  2001)  which  intends  to  certify  the  relationship  between  the  predictor  variable  of 
moderating  levels  and  outcome  variables  of  student  engagement. 

Data  Collection 

The  prime  data  source  for  this  study  will  be  the  automatically  archived  conference  transcripts  from  an 
online  three-credit  course  offered  at  a Canadian  University.  This  course  on  interpersonal  communication  is 
delivered  through  a real-time,  interactive  text,  image,  and  animation  messaging  system  and  it  is  one  of  the  first 
synchronous  technologies  that  offer  a visual  representation  of  participant  interaction.  This  course  is  unique  also 
in  the  respect  that  all  activities  and  interactions  happen  in  real-time,  i.e.  in  synchronous  mode  (Lobel,  et  al., 
2002). 

Though  the  transcripts  form  the  main  data  for  the  study  we  plan  to  collect  other  data  to  triangulate  the 
results  (Patton,  2002).  These  additional  data  sources  include  course  syllabus;  course  readings;  classroom 
activity  agenda  developed  by  the  teaching  team  and  delivered  to  each  teaching  staff  two  days  before  class  once 
a week;  class  preparation — one  hour  online  meeting  of  the  teaching  staff  immediately  before  class,  and  course 
assignments.  These  data  will  help  better  understand  the  context  of  each  conference. 

Data  Analysis 

The  predicator  variable  of  the  study  is  moderating  level.  By  adapting  and  combining  Xin’s  (2002) 
rubric  for  measuring  online  moderating  with  Anderson  et  al.’s  (2001)  teaching  presence  model,  this  researcher 
has  created  a five-level  rubric  to  measure  the  moderating  level.  In  this  model,  the  minimal  level  of  moderating 
(level  1)  includes  when  the  moderator;  opens  discussion,  establishes  the  computer  conferencing  agenda,  and 
observes  conference  norms.  At  the  high  end  of  moderating  (Level  5),  the  moderator  strongly  weaves  and 
summarizes  participants’  ideas  in  addition  to  performing  the  previous  moderating  levels  or  functions.  Though 
termed  as  “levels,”  the  scale  embraces  both  the  quality  and  quantity  nature  of  moderating. 

The  outcome  variables  of  the  study  are  student  engagement  and  its  sub-constructs.  Student  engagement 
is  measured  through  three  indicators  (sub-constructs):  behavioral  engagement,  social-emotional  engagement, 
and  intellectual  engagement.  While  computer  log  data  provides  behavioral  engagement  information,  emotional 
engagement  is  assessed  through  emotional  expression  and  group  cohesion  attributed  to  closeness,  warmth, 
affiliation,  attraction,  and  openness  (Rourke  et  al,  1999).  Interactivity  and  higher-order  thinking  are  considered 
key  indicators  of  intellectual  engagement  in  this  inquiry.  In  terms  of  computer  conference  interactivity, 
declarative,  reactive,  and  interactive  messages  are  coded  (Hara  et  al.,  2000;  Henri,  1992;  Rafaeli  & Sudweek, 
1996;  Sarlin  et  al.,  2003).  In  terms  of  higher-order  thinking,  messages  of  problem  initiation,  problem 
exploration,  and  idea  integration  are  coded. 

Given  that  the  synchronous  conferencing  messages  are  relatively  short,  content  analyses  focus  on 
individual  message  units  as  the  unit  of  analysis.  A message  unit  is  considered  a posted  message  that  is 
automatically  numbered  by  the  system.  Inter-rater  reliability  (Krippendorf,  1980)  is  determined  using  Cohen's 
kappa. 


Preliminary  Findings  and  Discussion 

With  the  purpose  of  revealing  relationships  between  teacher  moderating  behaviors  and  student 
engagement  which  may  later  lead  to  the  articulation  of  a model  or  framework  for  online  teacher  moderation,  we 
have  conducted  preliminary  analyses  of  the  data  here.  To  date,  we  have  looked  at  the  relationship  between 
number  of  teacher  postings  and  student  attending  and  participating,  two  indicators  of  student  behavioral 
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engagement.  Second,  we  compared  the  outcome  variable  (student  engagement)  of  two  different  groups — one 
with  an  overall  high  moderating  level  (Level  5)  and  the  other  with  a somewhat  lower  moderating  level  (Level 
2). 

To  look  at  the  relationship  between  predictor  variable  teacher  posting  and  outcome  variables  attending 
and  participation,  a correlation  analysis  was  performed.  The  Pearson  correlation  between  teacher  posting  and 
student  attending  as  well  as  student  participation  were  significant,  respectively.  The  comparison  of  a portion  of 
the  outcome  variable  (student  engagement)  of  two  different  groups — one  with  an  overall  high  moderating  level 
(Level  5)  and  the  other  with  a somewhat  lower  moderating  level  (Level  2)  provided  interesting  findings. 
Findings  show  that  in  these  two  groups  behavioral  engagement  and  intellectual  engagement  are  about  the  same, 
whereas  emotional  engagement  is  at  least  two  to  three  times  higher  in  the  group  with  low  moderating  level. 
Further  analysis  of  one  indicator  of  intellectual  engagement — higher  order  thinking — shows  that  there  are 
minimal  differences  in  the  frequency  of  problem  initiation.  However,  the  level  of  exploration  is  higher  in  the 
group  with  a high  moderating  level  compared  to  the  group  with  a low  moderating  level.  In  contrast,  the  level  of 
idea  integration  is  higher  in  the  group  with  low  moderating  level  compared  to  the  group  with  high  moderating 
level. 

One  interesting  finding  that  emerged  was  the  striking  difference  in  the  number  of  creative  ideas  offered 
by  students  in  these  two  groups.  The  group  with  a high  moderating  level  only  produced  three  main  ideas  for 
their  project,  whereas  the  other  group  with  a low  moderating  level  produced  up  to  seven  different  ideas  for  their 
project.  Even  though  idea  generation  was  not  considered  intellectual  engagement  in  prior  research,  we  think 
these  differences  are  worth  exploring  in  the  other  online  classes.  We  are  interested  in  the  number  of  unique 
ideas  or  solutions  produced  by  group  members  since  it  is  a sign  of  creativity  and  divergent  thinking. 

Our  pre  liminary  analysis  of  a small  sample  of  the  data  indicates  some  interesting  trends.  As  we 
forecasted,  there  appear  to  be  a difference  in  student  engagement  levels  when  group  discussions  are  moderated 
at  different  levels.  However,  higher  levels  of  moderation  might  not  be  associated  with  higher  levels  of  student 
engagement.  Consequently,  we  believe  that  there  is  a need  to  discover  how  to  effectively  moderate  student 
behavioral,  emotional,  and  intellectual  engagement.  Higher  moderating  levels  might  be  more  conducive  for 
more  student  exploration,  but  may  interfere  with  idea  integration,  emotional  expression,  and  creative  ideas. 

Significance  of  the  Study 

This  study  explore  s student  engagement  in  relation  to  online  moderating  in  synchronous  computer 
conferencing.  Eventually,  research  in  this  area  can  extend  to  online  training  programs  and  curricula.  The  results 
of  the  study  could  allow  researchers  and  practitioners  develop  better  protocols  for  moderating  online 
discussions.  Such  knowledge  is  essential  if  online  learning  (particularly  synchronous  conferencing)  is  to  achieve 
its  full  potential. 
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The  Impact  of  Spoken  Instructions  on  Learner  Behavior  Following 
Multimedia  Tutorial  Instruction 

Howard  Solomon 
Florida  State  University 

Abstract 

The  choice  of  what  to  include  in  educational  software  is  an  issue  with  which  instructional  designers 
are  regularly  concerned.  Multimedia  capacity,  standard  on  today's  desktop  computers,  gives  designers  the 
opportunity  to  provide  learners  a more  exciting  learning  experience  than  simply  looking,  clicking,  and  then 
looking  some  more.  One  feature  that  can  make  multimedia  software  different  than  such  conventional  media  as 
books  is  the  technological  capacity  to  include  sound.  This  capacity  is  already  available  to  educational  software 
developers  regardless  of  whether  they  are  producing  instruction  infixed  media  or  networked  form.  Recent 
interest  in  the  use  of  sound  to  enhance  learning  ( Bishop  and  Cates,  2001a)  suggests,  however,  that  although 
there  is  reason  to  believe  that  sound  can  enhance  learning,  that  sound  is  used  infrequently. 

One  reason  given  for  the  scarcity  of  software  that  incorporates  sound  is  a lack  of  experimental 
evidence  indicating  a significant  learning  improvement  when  sound  is  present.  While  some  experimental 
evidence  supports  combining  sound  with  other  media  (Lai,  2000;  Moreno  & Mayer,  2000;  Nocente,  1996),  few 
experiments  have  directly  compared  textual  and  spoken  presentations  while  maintaining  content  and  visual 
information.  For  instructional  designers  seeking  to  accommodate  different  learning  preferences  by  offering 
products  that  facilitate  multiple  learning  modalities,  or  for  those  seeking  to  accommodate  learners  with  reading 
or  visual  impairment,  it  makes  sense  to  examine  whether  learning  by  listening  can  match  or  surpass  learning  by 
reading. 

This  paper  presents  a different  point  of  view.  Instead  of  looking  for  significant  learning  improvements 
in  the  presence  of  sound,  it  looks  instead  at  differences  in  the  ways  learners  behave  when  sound  is  introduced. 
With  this  kind  of  information,  it  becomes  possible  to  plan  to  take  advantage  of  the  strengths  of  sound  for 
accomplishing  differen  t parts  of  the  instructional  mission. 

Another  reason  that  sound  is  not  often  included  in  educational  software  is  that  it  simply  costs  more  in 
terms  of  time,  money,  and  computer  resources  to  include  it  than  to  leave  it  out.  Sound  is  an  investment  that  can 
only  weakly  point  to  learning  improvement,  and  then  usually  only  in  combination  with  other  interventions.  So, 
unless  it  can  be  shown  that  sound  can  lead  to  some  kind  of  improvement  in  learning,  there  is  little  reason  to 
include  it. 

This  paper  investigates  the  effect  of  sound  as  a presentation  modality  on  the  way  students  review 
procedures  learned  at  the  moment  the  procedures  are  first  being  applied.  It  reports  on  a series  of  three 
experiments  conducted  at  various  times  in  the  one  and  a half  years  before  October  of 2004.  The  findings  of  each 
experiment  generated  the  questions  investigated  in  subsequent  experiments,  with  the  result  that  the  experiments 
form  a series.  The  outcome  of  the  series  is  that  there  appears  to  be  an  effect  on  student  reviewing  habits  that  is 
influenced  by  the  modality  in  which  the  review  material  is  presented.  This  takes  the  form  of  a preference  for 
listening  to  the  presentation  when  the  desire  to  complete  the  task  makes  it  important  to  know  how  it  is  done  and 
when  learning  on  the  first  time  through  was  incomplete. 

Introduction 

As  educators,  we'd  like  to  make  learning  more  likely  to  occur  for  our  students.  We  introduce 
educational  software  in  the  hope  that  it  will  be  instrumental  in  bringing  about  the  changes  that  we  identify  with 
learning  having  taken  place.  The  changes  frequently  take  the  form  of  increases  in  the  ability  to  provide  answers 
to  questions  about  some  aspect  or  aspects  of  the  subject  matter  of  the  instruction.  In  other  cases,  the  changes  we 
seek  to  bring  about  are  new  or  improved  proficiencies  in  the  performances  of  various  tasks.  When  the  learning 
we  desire  falls  into  this  latter  category,  it  makes  little  sense  to  measure  it  as  if  it  were  the  ability  to  answer 
questions.  If  we  wish  to  measure  the  ability  to  perform  procedures,  we  need  to  measure  how  well  or  frequently 
the  procedures  are  performed  after  the  learning  experience. 

When  we  teach  people  to  perform  tasks  in  the  context  of  using  computer  programs,  their  learning  is 
rightly  measured  by  setting  up  the  conditions  in  which  the  tasks  are  performed  and  turning  the  learners  loose  to 
do  what  they've  been  taught.  A suggestion  that  learning  hasn't  taken  place  because  the  learners  are  unable  to 
answer  some  questions  is  irrelevant  to  the  purpose  for  which  the  instruction  was  undertaken  in  the  first  place. 
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A perennial  discussion  in  Educational  Technology  classes  is  about  whether  media  choices  impact 
learning.  Richard  Clark's  (1981)  claim  that  the  media  that  delivers  content  has  no  influence  on  the  learning  of 
that  content  is  substantiated  by  a meta-analysis  that  is  based  on  a shortage  of  consistent  valid  studies  that,  in 
some  way,  show  that  any  one  delivery  medium  outperforms  any  other  delivery  medium  at  delivering  the  same 
content.  A set  of  studies  I conducted  comparing  a spoken  with  a textually  presented  lesson  is  subject  to  the  same 
criticism.  But  by  shifting  attention  away  from  learning  and  focusing  on  behaviors,  they  provide  a small  amount 
of  evidence  that  media  influences  how  students  act  in  the  light  of  learning  without  showing  that  they  learn  more 
from  one  medium  than  the  other. 

There  exists  with  any  procedural  learning  a brief  instant  in  which  the  learning  of  the  procedure  is 
completed  and  the  application  of  the  procedure  is  not  yet  tried.  This  is  a moment  in  which  we  hope  that  the 
learning  we  have  been  anticipating  has,  indeed,  been  produced.  If,  for  instance,  a learner  is  instructed  in  the 
business  of  tying  knots  and  is  then  handed  a rope,  the  time  interval  between  when  the  rope  is  introduced  and  the 
first  knot  is  tied  is  a moment  of  hopefulness.  What  the  learner  does  at  this  moment  is  an  indication  of  the 
effectiveness  of  what  we  have  put  forward  as  a learning  experience.  The  studies  I conducted  all  share  an 
examination  of  this  moment  after  the  textual  or  auditory  learning  experience  has  taken  place.  They  do  so  in  the 
belief  that  a difference  in  what  the  learner  does  at  this  point  is  as  surely  a difference  in  learning  as  a difference 
in  outcome  examination  scores. 


Experiment  1 

In  the  first  study,  each  of  29  educational  technology  undergraduate  students  completed  one  of  three 
versions  of  an  interactive  software  tutorial  before  attempting  to  apply  the  learned  procedure.  The  three 
treatments  differed  by  presenting  the  instructions  for  the  procedure  only  with  text,  only  with  spoken  audio,  or 
with  an  available  choice  between  text  and  audio.  At  the  end  of  the  lesson  about  recording  sentences  and 
arranging  them  so  that  clicking  a hidden  button  activated  them,  many  students  concluded  the  lesson  with  a sense 
of  having  forgotten  the  earlier  parts  of  the  procedure  while  learning  the  later  parts.  It  was  clear  that  being  able  to 
go  back  into  the  material  to  review  it  was  a feature  that  many  students  expected.  The  option  of  reviewing  was 
not  yet  included  in  the  design  of  the  software  at  the  time.  Because  the  students  were  intent  on  succeeding  with 
the  procedure,  they  followed  a path  to  review  the  material  by  reopening  the  tutorial  program.  They  were 
completely  unaware  that  a timer  was  running  in  the  original  tutorial  program  to  monitor  their  performance  over 
time.  When  they  re-opened  the  program,  they  reset  the  timer.  The  experiment,  for  all  practical  purposes,  was 
useless.  But  I didn't  know  this  until  I talked  with  the  students  a week  later. 

In  the  intervening  week,  analysis  of  the  data  pointed  in  the  direction  of  concluding  that  the  reading- 
only  treatment  resulted  in  a higher  number  of  procedure  completions  than  the  listening-only  treatment.  When  I 
approached  the  students  with  this  result,  students  in  the  listening-only  version  suggested  two  plausible 
explanations.  The  first  was  that  as  college  students,  they  were  in  a group  for  whom  reading  was  a well -executed 
strategy  for  learning.  They  felt  that  there  is  a natural  progression  from  listening  to  reading  modality  built  into 
the  structure  of  academic  institutions.  In  other  words,  college  admissions  and  retention  naturally  select  strong 
reading  skill  as  a trait. 

The  other  explanation  was  that  the  reading  version  students  appeared  to  have  mo  re  time.  This  was  not 
a part  of  the  design  of  the  experiment,  and  clearly  invalidated  any  conclusions  that  could  be  reached  about  how 
the  treatments  affected  performance.  But  it  appeared  that  many  more  of  the  students  randomly  assigned  to  learn 
by  reading  made  efforts  to  review  when  compared  with  those  randomly  assigned  to  learn  by  listening.  21 
students  were  in  these  two  groups.  The  other  8 were  in  the  switchable  version  of  the  software.  Of  1 1 students 
who  were  assigned  to  learn  by  listening,  only  2 initiated  procedures  to  review  the  program.  On  the  other  hand,  7 
of  the  10  assigned  to  learn  by  reading  initiated  procedures  to  review.  The  distribution  of  students  to  groups  and 
their  tendency  to  reset  the  timer  is  shown  in  table  1.  Chi-Square  analysis  of  data  related  to  this  phenomenon, 
and  shown  in  table  1,  indicates  that  the  likelihood  that  chance  could  account  for  this  difference  is  less  than  .05. 
(Chi-Square  = 5.47,  Asymp.  Sig.  = .019) 


Table  1:  Students  who  reset  the  timer  by  unanticipated  reviewing 


Version 

N 

Students  resetting  timer 

Anticipated  # resetting  timer 

Reading 

10 

7 

4.5 

Spoken 

11 

2 

4.5 

Switchable 

8 

NA 
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There  was  no  reason  not  to  notice  that  while  students  in  either  treatment  group  had  equal  opportunity 
to  reset  the  program  timer  by  attempting  to  review,  nearly  all  students  who  did  so  had  experienced  the  reading 
modality  treatment.  The  ability  to  review  by  going  back  to  the  presented  text  is  a natural  affordance  of  reading  - 
something  we  expect  to  be  able  to  do.  Although  we  have  the  technological  means  to  do  so,  when  we  hear  a 
speaker  talk,  we  do  not  expect  to  be  able  to  recapture  what  we  heard  in  other  instances.  This  is  a major 
difference  between  a temporal  medium  like  speech  and  a non-temporal  medium  like  printed  text . The  use  of  a 
non-temporal  medium  leads  to  a stronger  expectation  that  reviewing  is  supposed  to  be  allowed. 

Experiment  2 

The  second  experiment  arose  from  the  first  explanation  that  college  students  were  a selected  group  of 
strong  readers  who  would  naturally  favor  learning  by  reading.  It  sought  to  investigate  the  effect  reading  level 
had  on  learning  from  different  modalities.  Studies  conducted  by  the  US  Department  of  Education  in  the  1970s 
(Taylor,  1972)  established  the  general  guideline  that  children  favor  listening  over  reading  as  a learning  modality 
through  the  6th  grade.  Beyond  this  age,  high  levels  of  practice  with  the  encoding  and  decoding  of  text  make  text 
a more  efficient  learning  modality,  and  one  that  most  learners  prefer  because  they  can  learn  more  quickly  by 
reading  rather  than  by  listening.  Because  7th  graders  were  thought  to  be  near  the  transition  point  where  text 
becomes  the  dominant  modality,  I felt  that  they  were  likely  to  still  regard  learning  by  listening  in  a favorable 
light.  This  experiment  set  out  to  compare  modality  effects  in  two  populations,  one  of  college  undergraduate 
students  and  the  other  of  7th  graders.  The  expected  learning  superiority  of  undergraduates  showed  up  and  was 
significant.  But  what  was  being  measured  was  the  relative  advantage  of  the  listening  modality  between  groups. 
Although  the  7th  graders  measured  showed  a higher  mean  benefit  from  the  listening  modality  than  did  the 
undergraduates,  this  difference  failed  to  reach  a significant  level. 

The  software  used  for  this  experiment  had  been  modified  from  the  previous  version  to  keep  track  of 
reviewing  habits.  At  the  conclusion  of  the  tutorial,  the  user  was  informed  that  there  was  access  to  a way  to 
review  while  engaged  in  the  task.  This  way  to  review  exposed  both  the  listening  and  reading  modality  material 
to  the  learner.  The  reviewing  choices  offered  to  students  are  shown  in  figure  1. 
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Figure  2:  Reviewing  choices  offered  to  learners  at  the  conclusion  of  the  tutorial  lesson 
File 
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Learners  who  had  learned  by  reading  could  choose,  if  they  wished,  to  review  in  the  listening  mode. 
Similarly,  learners  in  the  listening  modality  treatment  group  could,  if  they  desired,  choose  to  review  by  reading 
their  previous  instruction  as  text  on  a screen.  Because  of  the  exposure  to  the  alternative  presentation  modality  in 
the  reviewing  mode,  the  rate  at  which  the  task  was  performed  after  the  tutorial  could  not  be  attributed  to  the 
mode  in  which  it  was  presented.  However,  the  entire  group  of  students  who  used  the  reviewing  feature  at  all, 
regardless  of  whether  they  were  part  of  the  7th  grade  or  the  undergraduate  population,  chose  to  review  the 
majority  of  the  time  in  the  listening  modality.  Data  related  to  this  phenomenon  is  shown  in  table  2.  The  choice 
to  do  most  of  the  reviewing  as  a listening  experience  was  significant  at  the  pre -determined  alpha  = .1  level. 


Table  2:  Reviewing  choices  made  by  reading  level 


Reading  Level 

N 

LM  Chosen 

RM  Chosen 

Chi-square 

Asymp.  Sig. 

Below  Median 

14 

10 

4 

2.571 

.109 

Above  Median 

18 

11 

7 

.889 

.346 

Total 

32 

21 

11 

3.125 

.077* 

* Significant  at  V <.10 


Experiment  3 

Because  of  a low  confidence  in  the  previous  finding,  I replicated  the  previous  study.  25  undergraduate 
students  enrolled  in  an  introductory  educational  technology  course  completed  a tutorial  designed  to  teach  them 
the  same  linear  label-making  procedure.  As  before,  at  the  completion  of  the  tutorial,  they  were  given  a scenario 
in  which  they  were  asked  to  complete  the  task  multiple  times.  As  they  began  the  task,  they  were  also  instructed 


756 


that  they  could  review  by  looking  at  the  original  program.  The  reviewing  menu  (Figure  1)  consisted  of  two  lists 
of  all  the  available  steps  in  the  procedure,  one  of  which  was  in  a column  indicating  that  they  could  re  -read  the 
step,  the  other  of  which  was  in  a column  indicating  that  they  could  listen  to  the  step.  So  a choice  of  reviewing 
presentation  mode  was  always  offered  to  students  at  the  time  they  needed  to  retrieve  procedure  information.  Of 
the  25  students  who  completed  the  project,  17  reviewed  while  engaged  in  the  task.  14  of  the  17  conducted  the 
majority  of  their  reviewing  of  the  procedure's  steps  in  the  listening  modality  (Chi-Square  = 7.118,  Asymp.  Sig. 
= .008).  This  result  is  shown  in  table  3. 


Table  3:  Favored  (majority)  reviewing  modality  used  by  student 


N 

Reading 

Listening 

Completed  Presentation  in  Switchable  Modality 

25 

Favored  for  Reviewing 

17 

3 

14 

Initially  Chosen  for  Reviewing 

17 

1 

16 

Because  the  learners  could  freely  choose  between  listening  and  reading  mode  presentations  while  they 
reviewed,  their  initial  choice  of  reviewing  modality  may  not  have  coincided  with  their  majority  choice.  The 
initial  choice  to  review  in  the  listening  modality  occurred  16  of  the  17  times  that  reviewing  occurred  (Chi- 
Square  =13.235,  Asymp.  Sig.  = .000).  Regardless  of  the  path  they  chose  for  going  through  the  tutorial,  at  the 
point  where  they  first  sought  help  before  beginning  to  apply  their  knowledge,  their  choice  was  consistently  to 
have  somebody  tell  them  how  something  is  done  rather  than  read  how  something  is  done. 

Discussion 

Does  the  media  have  an  impact  on  the  learning  experience?  Robert  Kozma  (1991),  in  responding  to 
Clark  claimed  that  media  are  an  integral  part  of  the  design  process.  He  elaborated  that  the  whole  instructional 
package  is  designed  to  include  specific  media  because  the  designers  anticipate  the  user  acting  and  reacting  with 
that  media.  Reviewing  at  the  beginning  of  the  task  is  a critical  point  where  the  interaction  of  the  user  and  the 
media  takes  place.  Hannifin  and  Land  (1997)  advocated  using  technology  to  give  students  choices  in  their 
learning  process.  The  choice  of  how  reviewing  material  is  to  be  presented  is  a choice  that  can  be  included  and 
uses  the  capacity  of  multimedia  to  facilitate  review. 

At  the  end  of  a tutorial  about  a computer  procedure,  the  best  guarantee  that  the  student  has  learned  the 
procedure  is  to  observe  the  procedure  being  put  into  action  without  further  guidance.  Unfortunately,  students  do 
not  always  end  the  tutorial  with  sufficient  confidence  to  expect  to  complete  the  procedure  successfully. 
Reminders  at  the  point  where  the  procedure  is  being  applied  can  be  helpful.  Such  reminders  can  take  may 
different  forms.  Job  aids,  for  instance,  exist  for  the  sake  of  reminding  performers  what  they  already  know  but 
may  not  immediately  remember. 

When  a computerized  tutorial  is  immediately  followed  by  a computerized  task,  the  learner  is  being 
asked  to  demonstrate  the  learning  in  the  exact  environment  where  the  learning  took  place.  This  is  why 
computerized  multimedia  learning  is  an  ideal  teaching  tool  for  teaching  computer-related  procedures.  Except  for 
possible  savings  of  computer  memory  - which  by  now  is  a non-issue  - there  is  little  reason  to  close  the  tutorial. 
Keeping  it  open  makes  it  convenient  as  a support  to  the  performance  just  learned.  In  effect,  the  learning  material 
switches  its  function  to  that  of  an  electronic  performance  support  tool.  And  at  the  point  where  this  switch 
occurs,  learners  consistently  choose  to  have  the  instruction  presented  to  them  in  auditory  form. 

Students  in  the  classes  concerned,  when  asked  for  their  opinions  about  why  these  results  were 
obtained,  offered  very  few  potential  answers.  Among  the  suggested  explanations  was  a greater  sense  of  security 
with  a speaking  voice.  We  ought  not  to  forget  that  language  is  a spoken  representation  of  the  phenomena 
observed  in  reality,  and  printed  text  is  an  abbreviated  form  of  language.  Text  gains  its  non-temporal  advantages 
at  a cost  of  leaving  out  inflection  and  accent.  It  appears,  from  the  results  obtained,  that  when  students  want 
critical  small  bits  of  the  information  about  a procedure  they  just  learned,  they  want  it  in  a way  that  delivers  as 
much  of  the  meaning  as  possible. 
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Abstract 

Interaction  is  one  of  the  key  variables  involved  in  successful  online  learning.  Previous  studies 
analyzing  online  interaction  are  limited  in  their  ability  to  reveal  dynamic  aspects  of  interaction  in  online 
learning  environments.  The  aim  of  this  study  is  to  develop  and  test  the  multi-level  assessment  framework  for 
analyzing  multi-facets  of  online  interaction.  To  achieve  this  purpose,  a multi-level  assessment  framework 
including  micro -assessment  and  macro  -assessment  was  proposed.  The  multi-level  assessment  framework,  then 
was  tested  with  the  data  collected  from  an  online  course.  This  study  suggests  that  multi-level  assessment 
framework  provides  us  with  different  kinds  of  information  that  helps  us  to  understanding  interaction  in  online 
environments. 


Introduction 

The  rapid  development,  in  recent  years,  of  Internet  technology  has  changed  the  nature  of  interaction  in 
online  environments.  As  a result,  there  is  raised  concern  over  how  to  assess  the  interaction  of  online 
environments.  Previous  studies  analyzing  online  interaction  have  focused  mainly  on  either  micro  -level  analysis 
or  macro-level  analysis.  The  micro -interaction  analysis  is  to  examine  the  content  of  the  information  acquired  in 
the  process  of  interaction  (Henri,  1991;  Offir  & Lev,  2000;  Oliver  & Mcloughlin,  1996).  The  macro  - interaction 
analysis,  on  the  other  hand,  is  to  examine  the  flow  or  patterns  of  interaction  (Henri,  1991;  Levin,  Kim,  & Riel. 
1990). 

Although  these  methods  provide  us  with  a tool  for  identifying  the  nature  of  the  interaction  occurring  in 
online  environments,  they  are  limited  in  their  ability  to  reveal  dynamic  aspects  of  interaction  in  such 
environments.  Each  analysis  method  has  a limitation  in  which  it  does  not  completely  cover  the  dynamics  of 
online  interaction.  The  multi-facets  of  interaction  in  online  environments  require  us  to  build  a more 
comprehensive  assessment  framework  that  will  encompass  various  assessment  methods  proposed  in  previous 
studies.  Analyzing  interaction  at  both  micro  and  macro  levels  is  expected  to  provide  us  with  different  kinds  of 
information  that  helps  us  to  understand  interaction  in  online  environments. 

Therefore,  the  purpose  of  this  article  is  to  propose  and  test  a multi-level  assessment  framework  for 
analyzing  interaction  in  online  environments.  To  achieve  this  purpose,  we  first  propose  the  multi-level 
assessment  framework  for  analyzing  online  interaction.  Next,  we  evaluate  the  multi-level  assessment 
framework  with  the  data  collected  from  an  online  course. 

Multi-Level  Assessment  Framework  for  Analyzing  Online  Interaction 

An  initial  study  proposed  a systemic  assessment  framework  to  analyze  online  interaction.  Song  (2003) 
proposed  a multi-level  assessment  framework  to  analyze  various  aspects  of  interaction  in  online  environments. 
According  to  his  framework,  the  unit  of  analysis  at  the  micro  level  is  the  individual  message.  The  content 
analysis  of  individual  messages  uncovers  the  nature  of  shared  information.  As  a result  of  this  microanalysis, 
each  individual  message  can  be  divided  basically  into  two  main  dimensions;  cognitive  and  social.  The 
dimensions  can  again  be  divided  into  more  specific  sub-dimensions  according  to  the  nature  of  the  interaction 
between  participants.  At  the  macro  level,  the  unit  of  analysis  is  multiple  messages.  Since  individual  messages 
are  related  to  other  messages,  they  form  a multiple  message  combining  those  individual  messages.  Therefore, 
macro -analysis  gives  a good  initial  approach  to  sketching  out  the  big  picture  of  an  interaction  and  can  serve  as  a 
way  of  identifying  messages  to  be  analyzed  more  deeply.  It  also  gives  a context  to  the  individual  messages  as 
they  are  analyzed  at  the  micro  level. 

However,  this  framework  had  a limitation  in  that  it  did  not  provide  specific  guidelines  in  its 
application.  Therefore,  one  of  the  goals  of  this  current  study  was  to  investigate  whether  the  framework  would  be 
useful  to  analyze  online  interaction.  While  testing  the  framework,  we  identified  critical  categories  and 
assessment  criteria  for  each  category.  Tabe  1 shows  multi- level  assessment  framework  that  includes  assessment 
levels,  assessment  units,  assessment  categories,  and  assessment  criteria  for  analyzing  online  environments. 


759 


Table.  1 Assessment  Levels,  Assessment  Units,  Assessment  Categories,  and  Assessment  Criteria 


Level 

Analysis  Unit 

Analysis  Category 

Analysis  Criteria 

Micro 

Individual 

message 

• Cognitive  dimension 

• One  way  communication 

• Process  (procedure,  expository,  explanatory) 

• Cognitive  trust 

• Social  dimension 

• Two  way  communication 

• Process  (development  of  identity, 

development  of  social  climate,  and  managing 
the  affective  factors) 

• Social  trust 

Macro 

Multiple 

messages 

• Patterns  of  Interaction 

• Work  patterns  (dropouts,  crammers,  regular 
periods) 

• Type  of  participation  (lurker,  persistent) 

• Depth  of  interaction 

• Number  of  threads 

• Numbers  of  participants  per  thread 

• Length  of  posting 

• Purpose  of  interaction 

• Introduction 

• Clarify,  question,  elaborate 

• Conclude 

Analyzing  interaction  at  the  micro-level,  there  are  two  clearly  identified  dimensions  in  the  literature: 
cognitive  and  social.  We  define  the  cognitive  dimension  as  the  processing  and  organization  of  information  in 
order  to  create  knowledge  while  the  social  dimension  as  social  presence  including  affective  elements  to  create  a 
community  of  learning.  In  addition  to  these  two  broader  dimensions,  we  have  identified  three  categories  of  sub- 
dimensions that  can  be  used  for  both  cognitive  and  social  interaction  research:  type  of  communication, 
processes,  and  trust.  First,  in  comparing  cognitive  and  social  interaction,  there  is  a difference  in  the  way  that  on- 
line participants  communicate.  Another  subdimension  that  can  be  studied  in  assessing  cognitive  and  social 
dimensions  of  on-line  interaction  is  the  processes  used  to  create  knowledge  and  a community  of  learning.  The 
most  commonly  used  categories  for  the  cognitive  dimension  are  Gunawardena,  Lowe,  & Anderson’s 
procedural,  expository,  and  explanatory.  While  Gunawardena,  Lowe,  & Anderson  framework  is  often  used  to 
categorize  cognitive  processes,  there  is  no  corresponding  framework  accepted  by  experts.  In  fact, 

Gunawardena,  Lowe,  & Anderson  model  does  include  a category  of  “social  processes’’  in  addition  to 
procedural,  expository,  and  explanatory.  However,  we  have  identified  corresponding  categories  that  are  the 
elements  in  creating  learning  communities:  development  of  identities,  developing  social  climate,  and  managing 
the  affective  factors.  The  last  subdimension  we  have  identified  is  cognitive  and  social  trust.  Kanawattanachai 
and  Yoo  (2002)  identify  the  factors  that  establish  cognitive  trust  as  reduction  of  complexity,  reliability  of 
functions  and  information  exchange,  and  delivery  of  promised  action  and  information.  Social  trust  is  those 
factors  such  as  empathy,  emotional  support,  and  free  expression,  that  are  the  basis  for  social  interaction  and 
relationship  building. 

The  macro  analysis  looks  at  long  term  participation  and  interaction  to  identify  patterns  and  trends  in 
on-line  learning.  Based  on  the  literature  and  our  own  analysis,  we  developed  three  main  areas  for 
macroanalysis:  patterns  of  interaction,  depth  of  interaction,  and  purpose  of  interaction  (DeLaat,  2001;  Garrison 
& Anderson,  2003;  Gunawardena,  Lowe,  and  Anderson,  1997;  Henri,  1991;  Hwang  & Wang,  2004;  Mazur, 
2004).  While  each  area  of  the  macro  analysis  has  its  own  characteristics,  it  is  important  to  look  at  how  each 
affects  the  overall  pattern  of  interaction.  It  is  the  combination  of  factors  that  researchers  need  to  look  at  when 
determining  what  is  happening  at  the  macro  level  (Howell-Richardson  & Mellar,  1996). 
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The  Contexts  for  Evaluation  of  Multi-Level  Assessment  Framework 


Participants 

Multi- level-assessment  framework  for  analyzing  interaction  in  online  environments  was  evaluated  with 
the  data  collected  from  an  online  course.  The  participants  in  this  study  were  twenty  graduate  students  who 
taking  an  online  course  in  Educational  Technology  program  at  a major  university  located  in  the  northeastern 
United  States. 

Data  Collection  and  Analysis  Procedures 

All  participants  took  part  in  an  online  group  discussion.  The  online  group  discussion  was  conducted  as 
an  extra-credit  activity  for  the  course  and  was  facilitated  via  a web-based  bulletin  board,  “Web  City”.  The 
discussion  took  place  over  a one  week  discussion  period.  Participants  were  randomly  assigned  to  two  different 
types  of  group  discussions:  an  instructor-moderated  discussion  or  a student-moderated  discussion.  The 
discussion  case  involved  Internet  security  issues:  “Why  privacy  and  security  of  personal  information  on  the 
Internet  became  an  increasingly  important  issue?  Who  is  responsible  to  protect  children's  privacy  and  security? 
What  can  we  do  in  order  to  protect  children's  privacy  and  security  in  classroom?  ”,  Students  in  a student- 
moderated discussion  group  discussed  and  created  roles  related  to  the  problem  case,  chose  their  roles,  discussed 
the  problem  case  with  their  group  members,  and  completed  a discussion.  On  the  other  hand,  the  students  in  the 
instructor-moderated  discussion  group  followed  the  same  procedure,  except  that  they  did  not  create  and  select  a 
role.  Data  were  analyzed  using  the  categories  of  the  multi-level  assessment  framework  developed  in  this  study. 

Results 

Micro-Level  Analysis 

Cognitive  Dimension 

Communication  Type.  For  the  most  part,  communication  in  the  cognitive  dimension  is  a one-way 
transmittal  between  the  sender  and  receiver  (Chen,  Wong,  Hsu,  2003).  Interaction  between  teacher  and  student, 
for  example,  often  falls  into  the  cognitive  dimension.  As  the  example  below  illustrates,  the  instructor  does  not 
expect  any  feedback  from  the  student  on  the  content  of  the  information  unless  the  student  has  questions.  This  is 
teacher  initiated  with  the  expectation  that  further  interaction  will  need  to  come  from  the  instructor.  For  example, 
in  the  on-line  discussions,  the  instructor  wrote: 

The  main  task  in  today  (Monday)  and  tomorrow  (Tuesday)  is  to  read  the  problem  case  and  create 
three  important  roles  that  we  will  play.  To  create  the  roles,  l hope  you  find  answers  to  the  following 
questions:...  [Posts  four  questions]...  Please  post  a note(s)  in  which  you  show  a role  that  you  feel  most 
important  to  solve  this  problem  before  Tuesday. 

It  is  clear  that  this  is  one-way  communication  with  the  instructor  initiating  the  discussion  and  the 
student  expected  simply  to  respond.  Chen,  Wong,  and  Hsu  (2003)  associated  this  dimension  with  strictly 
teacher-student  interaction.  However,  student-student  interaction  can  also  have  this  cognitive  dimension.  In  the 
following  example,  the  student,  like  the  instructor  above,  initiates  the  interaction  and  does  not  expect  any 
feedback  from  the  other  group  members.  Any  further  communication  about  the  information  provided  will  need 
to  be  initiated  by  the  student  that  posted  the  paper. 

The  problem  is  how  can  we  provide  for  children ’s  privacy  on  the  Internet  that  is  effective,  cost 
effective,  and  reliable?  The  answer  is  very  complex  and  I am  attaching  a paper  about  this. 

Process  of  Interaction.  First,  the  procedural  interaction  can  take  the  form  of  orienting,  subdividing  the 
problem,  and  designing  (Gay  & Lentini,  1995),  task  description,  scheduling,  or  identifying  tools  and  techniques 
to  be  used  (McFadzean  & McKenzie),  goal  and  objective  creation  (Owen,  Pollard,  Kilpatrick,  and  Rumley, 
1998)  or  teacher  designed  framework  within  which  students  are  expected  to  work  (Moaliem,  2003,  Thorpe, 
2002).  In  our  research,  the  instructor  gave  students  a general  framework  on  tasks  that  needed  to  be 
accomplished  and  a schedule  of  due  dates.  Throughout  the  course,  at  key  points  when  tasks  needed  to  be 
accomplished,  he  would  give  further  directions  on  how  to  accomplish  a certain  task.  Students  also  set  their  own 
schedules  and  procedures  within  the  course  such  as: 

7 have  posted  a resonse  to  ( Student  Name ) ’sessay  that  bridges  from  the  material  that  I posted 
yesterday.  I think  that  you  will  find  my  opinions  and  views  there,  as  well  as  in  the  framing  remarks 
that  1 made  before  posing  some  of  the  illunstrations  from  the  AOL  model.  ..If  you  have  ideas  as  to  how 
to  compile  our  team  report,  l am  open  to  them.  One  way  to  compile  a team  report  might  be  to  thread 
them  with  one  another  here.  Or  perhaps  we  open  another  thread  to  do  this.  This  is  new  ground  that 
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we  are  exploring  so  I think  that  we  will  have  to  figure  out  what  we  are  doing  together  with  this  as  we 
are  doing  it. 

Another  component  to  processes  is  expository  interaction.  Expository  interaction  can  include 
organization  of  information,  concepts  put  into  different  contexts,  or  representation  of  knowledge  (Cunningham- 
Atkins,  Powell,  Moore,  Hobbs,  Sharpe,  2004),  evaluation/application,  problem  identification,  definition, 
exploration  (Newman,  Webb,  & Cochrane,  1992),  moving  from  concrete  to  abstract  ideas  (McMahen,  1995),  or 
summarizing  concepts  (McFadzean  & McKenzie,  2001).  An  example  of  this  is  an  instructor  posing  questions  in 
order  to  illicit  student  ideas  or  information  such  as  the  sample  above  when  the  instructor  posed  4 questions  for 
the  students  to  answer.  Another  example  of  this  is  a student  posting  answers  to  a question.  The  interaction  in 
this  component  focuses  on  the  content  and  usually  is  a result  of  well-structured  instructional  design  (Moallen, 
2003;  Thorpe,  2002). 

Finally,  the  explanatory  interaction,  often  (although  not  always)  is  a response  to  questions  raised 
during  expository  interaction.  Often  it  is  the  instructor  that  will  provide  explanation,  although  other  students 
can  fill  this  role,  especially  in  smaller  group  activities.  Explanation  may  require  negotiating  understanding 
(Gay  & Lentini,  1995),  integration  or  general  clarification,  in-depth  clarification,  inference,  judgment 
(Newman,  Webb,  & Cochrane,  1992)  or  determining  the  information  gap  and  what  is  needed  to  close  it 
(Kay worth  & Leidner,  2001/2002).  In  the  case  of  our  research,  the  instructor  used  explanatory  interaction  to 
clarify  tasks  and  goals  in  accomplishing  tasks.  However,  he  also  summarized  key  points  at  the  end  of  the  tasks, 
which  acted  to  explain  the  issues  and  concepts  presented  in  the  course.  While  this  is  an  obvious  example  of 
explanatory  interaction,  student  explanations  also  help  in  the  developing  understanding.  For  example, 

I need  to  offer  a disclaimer.  The  material  that  I posted  to  get  things  going  yesterday  were  elements  of 
AOL’s  way  of  responding  to  these  issues.  AOL  has  certainly  been  very  successful  as  a business  in 
general  and  as  one  which  must  be  attentive  to  these  privacy  and  security  issues. 

In  this  case,  the  student  was  offering  further  insight  into  readings  he  had  posted  on  the  previous  day. 

He  felt  that  the  student  he  was  writing  to  may  not  have  understood  the  basis  of  the  information,  based  on  her 
response  to  his  original  posting. 

Cognitive  Trust.  Cognitive  interaction  needs  information  and  processes  that  are  reliable,  productive, 
and  relevant  to  the  task.  In  the  following  case,  the  student  is  questioning  the  reliability  of  the  information  on 
which  she  needs  to  base  a paper.  She  also  has  identified  a flaw  in  the  process,  an  indication  of  lack  of  cognitive 
trust. 

I am  not  sure  if  I can  write  the  final  paper  because  I feel  there  is  information  missing  from  all  groups. 

I know  that  there  is  information  missing  from  the  parents  group  because  at  least  one  of  { student 
name]  ’s  and  one  of  my  postings  wasn’t  in  the  summary. 


Social  Dimension 

Communication  Type.  In  developing  a learning  community,  the  interaction  tends  to  be  two-way 
(Chen,  Wong,  Hsu,  2003).  The  receiver  is  expected  to  contribute  equally  to  the  communication  process  as  the 
sender,  initiating  feedback  even  when  it  is  not  elicited.  The  level  of  communication  is  deeper  and  more 
complex,  as  defined  by  Henri  (1991).  Communication  initiated  by  the  student  to  the  instructor  can  be  an 
example  of  the  social  dimension.  The  following  example  demonstrates  an  equality  in  the  initiation  of  ideas: 
After  posting  the  responses  to  a questionnaire : To  tell  the  truth,  while  l didn  't  quite  follow  your 
schedules,  I still  think  that  your  initial  moderation  strategy  is  pretty  impressive  and  interesting. 
Personally,  I think  that  if  this  task  was  not  scheduled  almost  toward  the  end  of  this  semester,  we  should 
have  been  able  to  proceed  this  discussion  with  more  fun.  Unfortunately,  at  this  point  of  time,  we  have 
many  homework  or  projects  due,  which  distracts  us  from  discussion  more  or  less.  Isn’t  it? 

One  limitation  to  using  the  direction  of  communication  is  that  it  excludes  content  and  purpose  of  the 
interaction.  In  fact,  some  two-way  communication  can  be  cognitive  and  some  one-way  communication  can 
indicate  social  interaction.  As  a result,  assessment  of  this  subdimension  alone  does  not  indicate  one  type  of 
dimension,  but  rather  it  helps  identify  the  nature  of  the  interaction. 

Process  of  Interaction.  The  first  subdimension,  developing  identities,  focuses  on  creating  a group 
identity.  In  creating  a group  identity,  members  explicitly  or  implicitly  determine  the  norms  of  group  behavior, 
interaction  conventions,  shared  language,  and  identification  with  the  community  (Kling  & Courtright,  2003; 
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McFadzean  & McKenzie,  2001;  Owen,  Pollard,  Kilpatrick,  & Rumley,  1998).  This  is  achieved  through  group 
involvement,  creating  feelings  of  contentment  and  belonging,  and  creating  new  symbolic  convention  and 
communication  codes(Oren,  Mioduser,  Nachmias,  2002).  In  the  data  we  analyzed,  group  members  began  their 
e-mails  by  greeting  others  with  their  group  name,  e.g.“hi  government  group”.  This  clearly  identified  the  group 
members  with  the  ‘‘government”  group,  establishing  membership  through  the  team  name.  Another  example  of 
establishing  identity  was  the  student  that  referred  to  “we”  in  stating  the  group’s  position.  By  using  “we”  and 
“us”  she  is  identifying  two  groups  that  have  different  membership  requirements.  Not  everyone  can  be  “us”, 
only  those  in  the  business  group. Part  of  the  business  group,  she  wrote: 

Would  you  expect  us  to  comply  with  those  regulations  or  even  contribute  to  this  community’s  growth 
once  we  are  out  of  business? 

The  second  aspect  of  social  processes,  developing  social  climate,  focuses  on  creating  the  behavioral 
norms,  shared  language,  and  interaction  conventions  of  the  group  (Kling  & Courtright,  2003;  Owen,  Pollard, 
Kilpatrick,  and  Rumley,  1998).  As  a community  is  created,  members  need  reassurance,  trust  (Gay  & Lentini, 
1995),  feedback,  advice,  and  support  (McFadzean  & McKenzie,  2001).  This  is  often  achieved  through 
socializing  and  the  exchange  of  personal  information  (Moaliem,  2003;  Rovai,  2002;  Vissar,  Plornp,  Amirault, 
Kuiper,  2002).  This  socializing  helps  to  develop  group  norms  and  values  that  are  the  basis  for  the  on-line 
community.  The  social  climate  can  be  negative  or  positive.  It  is  obvious  that  student  to  student  interaction 
makes  up  a large  part  of  this  aspect  of  social  process.  For  example,  one  student  summarizing  another’s  ideas 
ends  with,  “ hopefully  I’m  representing  you  correctly!  If  not,  just  add  at  will.”  Social  climate  may  need  to  be 
moderated  by  the  instructor.  In  fact,  instructor  interaction  may  set  the  tone  for  the  social  climate.  For  example, 
by  beginning  instructions  for  student  tasks  with  “I  am  so  happy  to  work  with  you”,  the  instructor  is  setting  the 
tone  for  social  interaction  expectations.  In  other  words,  all  group  members  should  be  open  to  working  with 
others  in  their  group.  The  instructor  is  also  implying  that  work  is  a collaborative  effort,  not  a transfer  of 
knowledge  from  instructor  to  student.  He  clearly  states  that  he  is  working  with  the  students,  not  that  the 
students  are  doing  work  for  the  instructor. 

The  last  aspect  of  creating  a community  is  recognizing  and  managing  the  affective  factors  that 
students,  the  group,  and  even  the  instructor  have.  This  may  include  participant  emotions  and  feelings  (Vissar, 
Plomp,  Amirault,  & Kuiper,  2002),  motivation,  conflict  and  conflict  resolution,  student  control  and  efficacy 
(Moaliem,  2003),  tolerance  of  ambiguity,  and  comfort  with  technology.  This  is  the  hardest  subdimension  to 
identify,  since  affect  is  not  always  expressed  explicitly.  In  face-to-face  interactions,  participants  can  see  non- 
verbal social  cues  such  as  body  language,  eye  contact,  and  changes  in  the  voice.  On-line  interaction  does  have 
non-explicit  social  cues,  however.  For  example,  the  student  in  the  following  posting: 

Many  of  the  teacher  concerns  are  similar  to  the  parental  concerns,  because  the  teacher  can  give  online 
consent  for  the  parent  in  a school  setting. 

Social  Trust.  Social  trust  is  based  on  empathy,  free  expression  and  emotional  bonds  (Kanawattanachai 
& Yoo,  2002).  These  are  the  affective  attributes  that  are  the  basis  for  community  and  the  creation  of  identity 
within  the  community.  Social  trust  can  affect  group  processes  (Hiltz,  1998),  group  loyalty,  motivation,  and 
shared  knowledge  creation.  One  major  cause  for  lack  of  social  trust  is  a lack  of  social  presence  by  both 
instructor  and/or  students.  The  following  is  an  example  of  how  social  trust  is  established: 

I leave  for  Kansas  City  at  6:25  cun.  Thursday,  arrive  back  in  Philly  at  11:45  p.m.,  then  take  my  dad  to 
Hopkins  for  an  appointment  in  Baltimore  at  8:30  a.m.  Firday.,.1  will  not  be  able  to  respond  to 
everyone ’s  posts  tomorrow,  in  other  words.  I will  check  in  and  post  on  Friday  evening. 

In  this  case,  the  student  is  explaining  his  lack  of  social  presence  for  the  next  few  days.  As  a result,  he  is 
creating  expectations,  which  establishes  social  trust.  He  also  gives  some  personal  information  which  can  create 
feelings  of  empathy  and  an  emotion  bond  with  the  group  members. 

Macro -Level  Analysis 

Patterns  of  Interaction 

Looking  at  the  posting  patterns,  who  posted  when,  we  could  get  an  idea  of  student  work  patterns.  As 
shown  in  the  Table  2,  we  found  that  one  group  did  not  post  during  last  three  days.  Although  more  than  one 
week’s  worth  of  data  would  be  necessary  to  determine  trends,  looking  at  postings  throughout  the  entire  course, 
we  could  determine  if  this  was  their  regular  work  pattern.  According  to  Hwang  & Wang  (2004),  there  are  three 
different  types  of  student  work  patterns:  dropouts,  crammers,  and  regular  periodic  reading  learners.  Dropouts 
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tend  to  have  bursts  of  activity  at  the  beginning  of  an  on-line  activity,  then  stop  contributing  and/or  drop  out  of 
the  course.  Crammers  have  bursts  of  posting  activity  around  due  dates  or  at  the  end  of  an  activity,  not 
contributing  until  they  have  to.  Regular  periodic  reading  learners  may  post  regularly  or  may  have  multiple 
bursts  of  activity.  However,  they  develop  a sense  of  presence  throughout  the  activity.  By  identifying  student 
work  patterns,  researchers  can  test  various  instructional  designs  for  effectiveness,  identify  factors  that  enhance 
or  impede  participation,  and  determine  system  usage  rates.  As  shown  in  the  Table  below,  students  in  group  A 
seems  to  be  more  dropouts  where  as  those  in  group  B seems  to  be  more  regular  periodic  learners. 

Table  2.  Patterns  of  interaction 


Date 

Total 

Person 

3/24 

3/25 

3/26 

3/27 

3/28 

3/29 

3/30 

3/31 

4/1 

4/2 

4/3 

4/4 

4/5 

Group  A 

1 

6 

2 

8 

3 

4 

23 

2 

1 

1 

2 

3 

1 

1 

2 

1 

2 

7 

4 

1 

1 

2 

4 

5 

1 

1 

1 

1 

1 

5 

6 

1 

2 

3 

7 

2 

2 

1 

1 

1 

1 

1 

9 

8 

2 

2 

9 

2 

1 

1 

4 

Total 

6 

3 

12 

13 

4 

4 

8 

5 

3 

1 

59 

Group  B 

1 

5 

4 

4 

3 

7 

1 

2 

26 

10 

2 

1 

1 

1 

1 

6 

12 

2 

2 

13 

1 

3 

1 

1 

6 

14 

3 

1 

1 

5 

15 

2 

1 

1 

2 

1 

7 

16 

1 

2 

8 

3 

3 

17 

17 

1 

2 

3 

18 

1 

1 

2 

Total 

6 

12 

12 

16 

9 

8 

1 

6 

3 

1 

74 

Related  to  the  patterns  of  interaction  is  the  type  of  participant  in  the  interaction  process.  In  the  macro - 
level  analysis,  we  are  concerned  with  how  a participant  affects  the  interaction  process,  not  the  prescribed  role 
he/she  or  the  group  assigns  the  participants.  Mazur  (2004)  identifies  two  of  these  roles  as  the  Lurker  and  the 
Persistent.  The  Lurkers  may  not  make  their  presence  known  except  at  key  points  in  the  course  (such  as 
assignment  due  dates).  However,  their  postings  indicate  that  they  are  keeping  up  with  the  group  interaction. 
The  Persistent  contributes  despite  negative  or  no  feedback.  In  table  1,  Student  # 9 was  a Lurker.  There  was  no 
posting  from  March  27  to  April  1.  However,  in  reviewing  the  April  1 posting,  many  of  the  group’s  postings 
were  incorporated  into  the  student’s  final  comment.  This  indicates  that  the  student,  while  not  actively  posting, 
did  review  other  group  members’  contributions. 

Depth  of  Interaction 

The  example  in  Figure  1 demonstrates  how  multiple  threads  can  be  related.  Message  617  resulted  in  5 
direct  postings.  However,  two  additional  threads  were  indirectly  related  to  this  initial  message.  One  was  a 
group  discussion  without  instructor  direction  generated  as  a result  of  message  617;  the  other  was  a summary  of 
discussion  (appointment  of  group  roles)  as  a result  of  message  617.  The  breadth  of  discussion  in  this  case  was 
much  more  than  an  index  of  threads  would  indicate.  The  complexity  of  the  interaction  and  the  non-linear 
sequencing  are  more  accurately  illustrated  in  the  conversation  map.  Looking  at  the  lengths  of  the  postings  and 
the  time  between  postings  help  indicate  the  level  of  urgency  in  the  interaction  and  the  amount  of  thought  that 
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may  have  gone  on  in  posting.  For  example,  in  message  639  the  student  indicates  that  she  is  confused  about  the 
assignment.  The  message  is  short  (less  than  10  lines)  indicating  urgency.  Three  additional  messages  are 
exchanged  in  a direct  thread  within  the  next  24  hours.  Each  of  these  are  medium  or  short  messages.  Comparing 
this  with  the  related  thread  starting  with  message  660,  these  messages  are  longer  and  posted  with  more  time 
between  each  one.  The  depth  of  the  interaction  appears  to  be  deeper  with  message  660  than  639,  even  though 
both  have  the  same  number  of  postings.  On  the  other  hand,  message  639  appears  to  be  more  urgent,  with  a need 
for  a definite  conclusion. 

Purpose  of  Interaction 

In  order  to  understand  what  is  going  on  in  a sequence,  each  posting  should  be  characterized  as  to  what 
it  is  doing  in  the  interaction.  Mazur  (2004)  identifies  three  different  characterizations  of  a posting:  initiation, 
repair,  and  close.  She  further  defines  repair  as  including  clarification  and  elaboration.  We  would  add  to  this 
questioning.  In  initiation,  the  topic  thread  is  introduced.  Message  617,  for  example,  begins  the  discussion  by 
giving  instructions  on  what  the  assignment  is  and  how  students  are  expected  to  proceed  in  the  discussion.  In 
response  to  this,  some  students  elaborate  by  posting  their  answers  to  the  assignment  (messages  658,  620,  and 
626).  By  posting  the  assignment,  each  of  the  students  also  finishes  the  conversation/topic.  However,  message 
655  elaborates  by  postponing  the  assignment.  As  a result,  the  topic  is  picked  up  later  under  another  thread 
starting  with  message  660.  Clarification  is  usually  in  response  to  a question  or  problem.  In  message  639,  for 
example,  the  student  is  unsure  of  the  assignment  (resulting  in  a question).  The  instructor  clarifies  the 
assignment.  The  student  elaborates  on  the  assignment,  indicating  why  there  was  confusion.  The  instructor  then 
clarifies  the  source  of  the  confusion  and  finishes  the  topic.  In  reviewing  the  purpose  of  the  interaction,  the  end 
of  the  threads  should  have  a concluding  statement.  However,  when  a conclusion  is  missing,  this  is  an  indication 
that  either  the  topic  is  being  continued  somewhere  else  (as  with  message  655)  or  it  has  not  been  thoroughly 
discussed.  By  analyzing  the  sequencing  of  the  conversations,  researchers  can  determine  how  best  to  facilitate 
on-line  interaction,  interaction  motivation,  barriers  to  interaction,  and  timing  of  assignments  and  modules.  This 
type  of  analysis  also  allows  for  the  identification  of  related  threads  and  ideas  between  threads  (Henri,  1991; 
Howell -Richardson  & Mellar,  H.,  1996). 


Conclusion 

The  purpose  of  this  paper  was  to  develop  and  test  the  multi-assessment  framework  for  analyzing  the 
multi-facets  of  online  interaction.  The  macro  analysis  helps  researchers  to  identify  trends  and  generate  questions 
that  can  only  be  answered  in  a microanalysis.  On  the  other  hand,  the  microanalysis  needs  a context  outside  of 
the  message  that  only  a macro  analysis  can  provide. 

The  application  of  this  framework  has  provided  several  areas  for  further  studies.  First,  in  applying  the 
microanalysis  to  an  actual  on-line  discussbn,  we  discovered  that  it  was  difficult  in  some  group  interactions  to 
separate  the  cognitive  and  social  interaction.  Henri  (1991),  in  fact,  developed  four  categories,  social,  cognitive, 
metacognitive,  and  interaction.  Since  we  are  looking  at  all  interaction,  the  last  category  could  be  renamed  socio- 
cognitive  interaction.  Future  research  should  look  at  those  aspects  of  interaction,  especially  within  group 
processes,  where  there  is  both  social  and  cognitive  interaction  at  the  same  time.  Second,  The  macro  analysis  of 
on-line  interaction  must  include  various  factors  as:  technology  used,  the  interaction  environment  (synchronous 
vs.  asynchronous),  stated  goals  of  the  interaction  (topic,  field  of  study,  task),  and  events  or  unplanned 
interruptions  that  took  place  during  the  interaction  (Garrison  & Anderson,  2003;  Mazur,  2004).  This  description 
will  explain  any  constraints  or  limitations  in  the  macro-level  analysis.  Finally,  this  framework  needs  to  be 
tested  in  various  contexts  and  needs  to  establish  interrater  reliability. 
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Note  1.  Response  Code:  I=Initiate  E=Elaborate  Q=Question  C=Clarify  F=Finish  N=Not  related . Note  2.  Length  Code:  S<10  lines  M:  10-20  lines  L>20  lines 
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Abstract 

The  purpose  of  this  qualitative  case  study  is  to  study  the  organizational  culture  and  context  of  two  school 
sites  in  which  instructional  technology  (IT)  use  by  teachers  is  evident;  to  study  what  and  who  influences  individual 
teacher  preferences  toward  IT  use;  and  to  describe  the  relationship  of  Mary  Douglas ' ( 1 982,  1 989,  1 992 ) grid  and 
group  typology  in  the  decision  process  to  implement  IT  use  in  curricula. 

Statement  of  the  Problem 

Public  schools  have  increased  the  amount  of  technology  available  to  teachers,  assisted  in  increasing 
teachers’  familiarity  with  technology  through  training,  and  supported  and  encouraged  the  use  of  technology  through 
administrative  directives.  In  spite  of  these  efforts  to  increase  the  use  of  instructional  technology,  public  school 
teachers  are  not  using  instructional  technology  to  create  the  anywhere,  anytime  classroom  that  was  anticipated. 

While  schools  tend  to  make  instructional  technology  resources  available  to  all  who  seek  them,  many 
teachers  refuse  instructional  technology  for  a variety  of  reasons.  Rogers  (2000)  indicates  that  barriers  to  instructional 
technology  adoption  and  integration  are  found  in  both  internal  and  external  sources: 

Internal  barriers  may  be  summarized  as  “teacher  attitude”  and  “perceptions”  about  a technology,  in  addition 
to  a person’s  actual  competency  level  with  any  technology.  External  sources  include  the  availability  and 
accessibility  of  necessary  hardware  and  software,  the  presence  of  technical  personnel  and  institutional 
support,  and  a program  for  staff  development  and  skill  building.  Barriers  that  cross  internal  and  external 
sources  are  lack  of  time  and  funding  and  the  unique  culture  of  the  institution,  (p.  459) 

Rogers  (2000)  and  Chiero  (1997)  summarize  several  studies  of  teacher  barriers  to  instructional  technology 
adoption  and  point  out  that  different  studies  discover  similar  barriers.  Internal  barriers  (barriers  imposed  by  the 
teacher)  include  lack  of  time  (both  personal  and  release)  to  learn  to  use  instructional  technology  and  integrate 
instructional  technology,  lack  of  role  models  (others  in  the  school  site  using  instructional  technology),  lack  of  other 
models  for  using  instructional  technology,  and  teachers’  attitudes  about  instructional  technology.  Rogers  (2000)  and 
Chiero  (1997)  also  summarize  external  barriers  (barriers  teachers  view  as  imposed  upon  them)  as  lack  of  availability 
and  quality  of  hardware  and  software,  lack  of  time,  low  levels  of  funding,  low  institutional  support,  minimal  staff 
development,  uncertainty  that  instructional  technology  affects  student  learning,  and  lack  of  technology  support. 

In  addition  to  the  question  of  adoption  and  integration,  instructional  technology  represents  a change  in  the 
way  teachers  and  school  officials  operate.  Change  is  always  difficult  and  change  in  schools  often  seems  even  more 
difficult  than  in  other  areas.  Gruenert  (2000)  determined  that  the  culture  of  the  school  itself  is  one  major  factor  in 
promoting  change  within  schools: 

If  things  do  not  change  it  is  because  the  existing  culture  did  not  allow  it.  Understanding  what  culture  is  and 
what  it  does  allows  leaders  to  orchestrate  real  change.  Shaping  school  culture  is  not  for  the  timid  (Peterson 
& Deal,  1998)  and  it  takes  a long  time,  five  to  seven  years,  (p.  17) 

The  adoption  and  integration  of  instructional  technology  is  a fundamental  change  in  school  culture  and  the 
education  community.  Cohen  (1987)  comments  upon  change,  technology,  and  schools: 

Americans  have  celebrated  technology  as  a powerful  force  for  change  nearly  everywhere  in  social  life.  . . . 
Computers  are  only  the  latest  in  a long  line  of  mythologized  machines,  endowed  with  near-miraculous 
powers.  . . . Americans  are  fond  of  picturing  technology  as  a liberating  force  ....  Nearly  all  of  the  new 
technologies  pressed  on  schools,  from  books  to  microcomputers,  also  have  been  advertised  as  agents  of 
liberation.  They  would  change  education  by  making  students  less  dependent  upon  teachers,  and  by 
reducing  whole  class,  lock-step,  batch-processed  teaching  and  learning,  (p.  154) 

Teachers  may  be  reluctant  to  embrace  the  change  Cohen  comments  upon  simply  because  they  have  found 


769 


other  changes  in  the  past  to  be  a false-cure  for  that  which  ails  public  education.  If,  in  the  past,  teachers  were  quick  to 
adopt  innovative  changes  only  to  find  that  the  innovations  did  not  solve  day-to-day  problems,  why  then  should 
teachers  quickly  adopt  more  innovations?  For  many  teachers,  it  is  much  easier  to  address  known  problems  through 
known  solutions.  Unfortunately,  school  cultures  that  do  not  promote  change  erect  external  barriers  to  teachers’ 
adoption  of  innovations  such  as  the  integration  of  instructional  technology  in  the  classroom. 

Another  important  facet  of  the  problem  exhibits  itself  in  public  school  education:  with  school  budgets 
straining  under  the  weight  of  wide-scale  budget  cuts  and  decreased  state  funding,  instructional  technology  can 
provide  some  relief  by  allowing  teachers  new,  less  costly  ways  of  communicating  with  students  and  patrons.  Yet, 
teachers  cling  to  older,  less  efficient,  more  expensive  communication  tools.  Add  to  that  the  ever-increasing  emphasis 
placed  on  high-stakes  testing  as  mandated  by  the  federal  government’s  Leave  No  Child  Behind  Act,  and  instructional 
technology  quickly  loses  its  place  among  school  priorities.  Teachers  appear  reluctant  to  embrace  change. 

Moursund  (1997)  has  studied  computers  in  education  and  has  applied  decades  of  personal  involvement  in 
computing  and  the  field  of  education.  While  he  has  been  an  advocate  of  education  reform  and  continues  to  believe 
that  technology  will  have  a positive  effect  on  education,  his  views  are  tempered  by  experience: 

In  retrospect,  it  is  clear  that  I have  been  overly  optimistic.  Educational  systems  are  quite  resistant  to 
change.  Progress  has  not  occurred  as  fast  as  I had  thought  it  would.  Still,  considerable  progress  has 
occurred,  and  the  groundwork  has  been  laid  for  further  progress.  It  is  clear  to  me  that  we  are  just  at  the 
beginning  of  a number  of  major  changes  in  our  educational  system  that  will  occur  because  of  continuing 
progress  in  information  technology.  (Morsund,  1997,  Preface) 

Dexter,  Anderson,  and  Becker  (1999)  studied  teachers’  use  of  computers  and  studied  whether  or  not 
computers  were  an  impetus  for  change.  They  determined  that  teachers  used  computers  to  facilitate  change,  but  that 
computers  did  not  cause  change.  Rather,  teachers  cited  experiential  reflection,  continuing  education,  and  school 
culture  as  the  driving  force  for  change.  However,  not  all  teachers  see  instructional  technology  as  a positive  change. 
Goodson  and  Mangan  (1995)  found  that  social  studies  teachers  saw  computer  instruction  as  detracting  from  pure 
content  instruction. 

Why  then  do  a small  number  of  teachers  embrace  change  and  demonstrate  high-level  competence  and 
integration  of  instructional  technology  while  other  teachers  adamantly  refuse  to  adopt  or  even  experiment  with 
technology?  The  answer  lies  in  the  culture  of  the  school  site  itself. 

Purpose  of  the  Study 

The  purpose  of  this  qualitative  case  study  is  to  examine  the  organizational  culture  of  two  specific  school 
sites  in  which  instructional  technology  use  by  teachers  is  evident;  to  study  what  and  who  influences  individual 
teacher  preferences  toward  instructional  technology  use;  and  to  describe  the  relationship  of  Mary  Douglas'  ( 1982, 
1989,  1992)  grid  and  group  typology  in  the  decision  process  to  implement  instructional  technology.  Douglas’  grid 
and  group  typology  provides  the  theoretical  framework  for  this  study.  Developed  in  1982,  the  typology  has  been 
used  to  study,  decipher,  and  compare  various  social  contexts  in  educational  settings.  Harris  (1995)  established  that 
the  typology  is  useable  when  applied  to  selected  educational  contexts.  Stansberry  (2001)  applied  Douglas’  typology 
to  the  study  of  faculty  instructional  technology  preferences  in  higher  education.  While  this  study  is  modeled  after  the 
Stansberry  study,  this  study  will  focus  on  teacher  preferences  in  the  middle  school  environment. 

Research  Questions 

The  following  research  questions  are  addressed  in  this  study: 

1.  How  is  instructional  technology  used  in  classrooms  in  each  of  the  selected  schools? 

2.  In  what  ways  does  the  use  of  instructional  technology  reflect  grid/group  realities  in  each  of  the 
selected  schools? 

3.  What  other  realities  were  revealed  in  each  of  the  selected  schools? 

4.  Was  grid/group  helpful  in  understanding  differences  in  teachers'  instructional  technology  use  in 
the  selected  schools? 

Theoretical  Framework 

Douglas’  typology  is  appropriate  for  use  in  studying  the  length  and  breadth  of  social  settings.  Harris 
(1995)  found  that  “one  of  the  model’s  most  beneficial  aspects  is  its  holistic,  comprehensive  nature.  It  is  designed  to 
take  into  account  the  total  social  environment  and  individual  member  interrelationships  among  each  other  and  their 
context”  (p.  619). 

Douglas’  (1982)  typology  defines  both  the  individual  working  within  a socially  constructed  group  (grid) 
and  the  group  itself  (group)  (see  Figure  1 ).  In  this  study,  the  individual  is  the  teacher  working  within  the  socially 
constructed  group,  the  school  site.  Harris  (1995)  summarizes  the  grid  and  group  dimensions:  “Grid  refers  to  the 
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degree  to  which  an  individual’s  choices  are  constrained  within  a social  system  by  imposed  formal  prescriptions  such 
as  role  expectations,  rules,  and  procedures”  (p.  620).  “Group  represents  the  degree  to  which  people  value  collective 
relationships  and  are  committed  to  a social  unit  larger  than  the  individual”  (Harris,  1995,  p.  621).  In  short,  grid 
measures  the  amount  of  autonomy  an  individual  exercises  within  the  socially  constructed  organization;  group 
measures  the  amount  which  members  of  the  organization  value  the  organization  itself. 

Each  of  the  dimensions  is  measured  from  low  to  high  or  strong  to  weak  (Douglas,  1982).  In  a high  grid  or 
strong  grid  environment,  there  is  “an  explicit  set  of  institutional  classifications  that  regulate  individual  interactions 
and  restrain  their  autonomy”  (Harris,  1995 , p.  620-21).  In  a strong  grid  school  setting,  teachers  have  little  say  in 
matters  such  as  curriculum,  pedagogy,  and  operations.  In  a low  grid  or  weak  grid  environment,  “there  are  few 
distinctions  among  members;  individuals  are  esteemed  more  for  their  behavior  or  character  than  their  ascribed  role 
status”  (Harris,  1995,  p.  621).  Teachers  in  a weak  grid  school  have  much  personal  control  over  their  curriculum  and 
teaching  methodologies. 

According  to  Harris  (1995),  in  strong  group  settings  “there  are  specific  membership  criteria,  explicit 
pressures  to  consider  group  relationships,  and  the  survival  of  the  group  becomes  more  important  than  the  survival  of 
individual  members  within  it”  (p.  622).  Strong  group  schools  value  collaborative  work  among  all  members  of  the 
staff  and  work  at  creating  a single,  unified  school  site.  In  weak  group  settings,  “pressure  for  group-focused  activities 
and  relationships  is  relatively  weak”  (Harris,  1995,  p.  622).  Weak  group  schools  do  not  encourage  collaborative 
projects,  and  there  is  no  evidence  of  teachers  working  together  for  the  common  good  of  the  school  site. 

In  a variety  of  ways,  grid  and  group  is  useful  in  studying  school  culture  and  studying  how  teachers  interact 
with  that  culture.  Grid  and  group  provides  a framework  within  which  individual  teachers  can  be  plotted  on  a scale  of 
their  “individuation”  (Douglas,  1982,  p.  190)  and  school  sites  can  be  plotted  on  a scale  of  their  “social 
incorporation”  (Douglas,  1982,  p.  190). 


Methodology 

The  participants  in  this  study  include  selected  teachers  and  administrators  in  two  different  K-12  school 
districts  within  the  state  of  Oklahoma.  These  two  school  sites  were  selected  by  purposive  sampling  (Erlandson 
Harris,  Skipper  & Allen,  1993),  chosen  for  their  differing  perceived  organizational  cultures  and  differing 
organizational  contexts.  Oklahoma  State  University  College  of  Education  faculty  members  who  are  knowledgeable 
of  area  schools  were  consulted  in  this  process. 

Approval  from  each  school  site  was  provided  to  the  Oklahoma  State  University  Institutional  Review  Board. 
Letters  were  mailed  to  the  school  site  administrator  requesting  entry  and  included  a copy  of  the  Oklahoma  State 
University  Institutional  Review  Board  application  and  a signature -ready  memo  indicating  permission  to  study  the 
school  site. 

All  teachers  at  the  selected  school  sites  were  invited  to  participate  in  an  online  survey  designed  to  elicit 
information  to  assist  in  determining  the  grid  and  group  make  up  of  the  school  site.  The  survey  instrument  was 
developed  out  of  the  theoretical  framework  provided  by  Douglas  (1982,  1989,  1992).  The  survey  is  a product  of 
previous  grid  and  group  surveys  and  discussions  with  advisory  committee  members  Dr.  Ed  Harris  and  Dr.  Susan 
Stansberry,  researchers  who  have  studied  grid  and  group  typology.  The  survey  consisted  of  17  forced-choice 
statements  respondents  were  required  to  select  one  statement  of  the  pair  which  best  describes  their  school  site.  Each 
pair  of  statements  was  designed  to  elicit  teachers’  perceptions  of  some  aspect  of  their  school  site  as  it  relates  to  grid 
and  group  typology.  However,  the  language  of  grid  and  group  was  not  used;  the  language  for  the  statements  was 
drawn  from  the  vocabulary  public  school  teachers  would  use  in  discussing  their  own  social  environment. 

Following  completion  of  the  online  survey,  respondents  were  invited  to  participate  in  a face-to-face 
interview  designed  to  gather  more  in-depth  information  about  school  culture  and  teacher  use  of  instructional 
technology. 

Following  collection  of  the  survey  responses  and  interview  volunteer  information,  appointments  were 
scheduled  at  a mutually  convenient  time  between  the  interviewee  and  the  researcher.  All  interviews  took  place  in  the 
interviewees’  classroom  or  other  location  in  their  school  site.  Participants  in  the  audio  taped  interviews  were  be 
given  informed  consent  documentation  and  given  opportunity  to  review  the  document  prior  to  signing.  A copy  of  the 
research  report  was  made  available  upon  request  by  the  subject;  in  addition,  the  location  of  the  report  if  published  in 
any  journals  will  be  provided  to  the  interviewee.  Participants  were  informed  of  their  right  to  decline  participation  in 
any  part  of  the  study.  Participation  was  not  mandatory;  any  subject  who  chose  to  decline  could  do  so  freely. 
Anonymity  was  ensured  using  pseudonyms  given  to  all  participants  involved  in  this  study. 
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Rationale  for  Qualitative  Study 

The  final  decision  on  methodology  and  the  data  collection  process  is  always  based  upon  the  research 
questions  and  the  types  of  research  to  be  conducted.  In  order  to  conduct  a descriptive  case  study,  triangulation  of 
rich  data  sources  is  necessary  to  inform  the  questions  of  how  and  why  instructional  technology  is  being  used  as  it  is. 
Therefore,  employing  both  quantitative  and  qualitative  data  collection  and  analysis  methods  in  a mixed  method 
study  is  appropriate.  It  should  be  noted,  though,  that  the  methods  are  not  ‘evenly’  mixed.  Quantitatively,  the  study 
seeks  only  responses  from  school  site  teachers,  administrators,  and  instructional  technology  professionals  to  17 
forced-choice  pairs.  These  responses  assist  in  determining  where  on  Douglas’  grid  and  group  typology  the  school 
site  falls.  While  this  determination  is  vital  (it  informs  the  data  collected  through  interviews,  observations,  and 
document  analyses),  it  is  only  one  part  of  the  study  and  needs  to  be  coupled  with  the  rich,  deep  data  from  the 
qualitative  inquiry.  It  is  this  qualitative  data  and  the  fact  that  the  study  seeks  to  come  to  an  understanding  of  two 
schools  sites  and  the  use  of  instructional  technology  in  those  school  sites  that  prompts  the  use  of  a case  study 
methodology  to  guide  the  research.  Survey  instruments  are  but  one  of  many  data  sources  available  to  the  case  study 
researcher,  and  the  use  of  a survey  to  collect  large  amounts  of  customary  data  is  supported  by  Anderson  (1998). 

Mertens  (1998)  cites  Stake,  Yin,  and  the  U.S.  General  Accounting  Office  when  establishing  a definition  of 
case  study  research.  She  relies  heavily  on  Stake  in  establishing  that  case  study  is  defined  not  by  methodology,  but  by 
the  case  under  scrutiny.  Mertens  (1998)  says  “the  commonality  in  the  definitions  seems  to  focus  on  a particular 
instance  (object  or  case)  and  reaching  an  understanding  within  a complex  context’’  (p.  166).  This  study  seeks  to  do 
just  that:  come  to  an  understanding  of  middle  school  teachers’  use  of  instructional  technology  within  the  context  of 
Douglas’  grid  and  group  typology.  The  conclusions  drawn  will  aid  in  understanding  school  culture,  instructional 
technology  use,  and  the  perceptions  and  attitudes  about  instructional  technology  of  teachers,  administrators,  and 
instructional  technology  staff  members. 

Mertens  (1998)  supports  the  use  of  case  study  research  when  “the  focus  is  on  diversity  among, 
idiosyncrasies  of,  and  unique  qualities  exhibited  by  individuals’’  (p.  163).  In  this  study,  interview  questions  and 
observations  sought  how  teachers,  administrators,  and  instructional  technology  staff  members  perceive  instructional 
technology  and  instructional  technology  use  in  the  school  site;  how  they  use  technology  in  instructional  applications; 
and  their  attitudes  about  instructional  technology  use,  both  positive  and  negative.  Quantitative  methodologies  cannot 
get  at  the  richness  of  information  needed  to  fully  understand  these  perceptions,  uses,  and  attitudes.  Only  through 
close  contact  interviews  and  observations  can  these  details  emerge. 

Additionally,  Patton  (2002)  supports  the  use  of  case  study  when  one  of  three  conditions  is  present:  the  need 
to  understand  humanistic  values,  when  no  statistically  valid  and  reliable  instrument  is  available  to  measure  what  is 
being  studied,  or  as  an  adjunct  to  a quantitative  study.  This  final  reason  is  precisely  the  case  here:  the  brief  survey 
instrument  is  only  one  tool  used  in  determining  where  the  school  site  falls  on  the  grid/group  typology,  and  it  does 
not  collect  the  rich  data  about  the  use  of  instructional  technology  needed  to  come  to  an  understanding  of  how  the 
school  site’s  typology  affects  instructional  technology  use  within  the  school  site. 

Significance  of  the  Study 

This  study  of  teacher  use  of  instructional  technology  in  a public  school  can  be  important  for  several 
reasons.  First,  Douglas’  grid  and  group  framework  has  not  been  used  to  date  in  a K- 12  setting  to  explain  the 
variations  in  preferences  for  instructional  technology  use  and  from  a socio -cultural  context  perspective.  It  is 
therefore  necessary  to  test  this  framework’s  usefulness  in  this  setting  to  determine  its  theoretical  significance  for 
future  case  studies  regarding  teacher  instructional  technology  use. 

Second,  according  to  Rogers  (2000),  teachers’  attitudes  about  instructional  technology  and  their  attitudes 
about  their  school’s  support  for  instructional  technology  use  are  important  factors  in  the  decision  to  use  or  not  use 
instructional  technology.  The  need  to  describe  why  some  teachers  use  instructional  technology  can  only  be  served 
by  exploration  of  the  socio -cultural  contexts  within  which  preferences  are  defined  and  perceived. 

Third,  as  schools  continue  to  embrace  instructional  technology  in  all  facets  of  their  culture  and  practices,  an 
understanding  of  teacher  preferences  for  instructional  technology  use  will  be  beneficial.  Often,  when  schools 
consider  a shift  toward  instructional  technology,  they  focus  on  hardware  and  software  within  a specific  budget. 

These  are  necessary  considerations,  but  they  should  not  be  seen  as  more  important  than  a consideration  of  the 
teachers’  cultural  preferences  for  instructional  technology  use  (Tierney  & Damarin,  1998). 

Finally,  the  research  directly  benefits  the  research  participants.  Each  school  and  each  participant  will 
receive  information  regarding  the  outcome  of  the  study  and  how  the  study  applies  to  the  organizational  culture  of  the 
school  site.  With  this  information,  teachers,  staff  members,  and  administrators  will  better  understand  the 
organizational  culture  of  their  school  and  be  better  prepared  to  operate  within  that  culture. 
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Maple  Grove  Middle  School 

Maple  Grove  School  District  is  a technology-rich  environment  where  teachers  are  given  a number  of  tools 
that  they  may  choose  to  integrate  into  their  teaching.  Efforts  on  the  part  of  the  former  superintendent  have  created  a 
school  site  where  technology  is  taken  for  granted,  where  there  are  computers  in  every  classroom  everyday,  where 
network  access  is  not  an  issue,  and  time -shared  labs  are  a thing  of  the  past. 

There  is  centralized  control  of  resources  at  Maple  Grove,  and  acquisition  of  resources  is  not  a status 
symbol:  teachers  have  whatever  they  need  to  do  their  jobs  and  are  allowed  to  make  decisions  about  classroom 
teaching  without  interference  on  the  part  of  the  administration.  As  teachers  encounter  a hardware  or  software  need, 
they  bring  that  need  to  the  attention  of  the  administration  and  in  all  likelihood  the  administration  meets  that  need 
without  fanfare  or  reservation. 

With  the  exception  of  the  school’s  information  system,  PowerSchool,  there  are  no  longer  mandates  from 
the  administration  that  dictate  how  teachers  are  to  use  technology.  This  freedom  to  choose  is  one  of  the  major 
indicators  of  the  weak  grid  typology  at  work  at  Maple  Grove. 

Students  at  Maple  Grove  Middle  School,  too,  enjoy  an  environment  where  they  may  make  choices  about 
technology  based  upon  their  skills  and  desires.  Students  who  need  to  may  take  computers  home;  they  have  access  to 
them  throughout  the  school  day,  and  often  influence  the  use  of  technology  in  the  classroom.  Allowing  such  student 
choices  is  also  indicative  of  Maple  Grove’s  weak  grid  culture. 

Instructional  technology  is  pervasive  at  Maple  Grove.  Throughout  the  building,  there  is  evidence  of  the 
extent  to  which  Mr.  Fayette  went  to  in  order  to  provide  the  technology-rich  environment.  Yet,  teachers  do  not  work 
to  create  a strong  group  culture.  They  do  not  collaborate,  they  do  not  share  lesson  plans  or  materials,  and  they  do  not 
identify  with  Maple  Grove  much  outside  the  school  day.  They  live  out  of  district  and  tend  to  have  their  own 
networks  of  friends  outside  of  school. 

These  observations  indicate  that  Maple  Grove  Middle  School  is  a weak  grid/weak  group  culture  which 
Douglas  (1982)  calls  Individualistic.  Figure  2 illustrates  the  grid/group  typology  of  specific  Maple  Grove 
characteristics. 


Hill  wood  Junior  High  School 

The  strength  of  both  the  grid  and  group  aspects  of  Hillwood  Junior  High  are  evident  in  the  way  that  rules 
have  been  established  under  which  members  must  operate.  While  there  are  indications  of  weakening  of  both  grid 
and  group,  these  are  minor  influences  on  the  overall  culture.  In  general,  members  enjoy  the  group  to  which  they 
belong  and  identify  strongly  with  that  group.  They  are  quick  to  sacrifice  their  own  needs  for  the  better  good  of  the 
group  by  volunteering  their  time  to  help  when  that  help  is  needed  by  the  school. 

Additionally,  the  administration  at  Hillwood  may  impose  a number  of  the  rules  under  which  members 
operate,  but  at  the  same  time,  they  are  establishing  these  rules  for  the  good  of  the  group.  They  have  the  mission  of 
Hillwood  Junior  High  in  mind  when  they  work  to  spend  limited  funds  so  that  the  most  good  can  come  from  the  few 
dollars  they  have  to  spend. 

Douglas  (1982)  calls  a strong  grid,  strong  group  culture  Corporate.  Figure  Three  illustrates  the  grid/group 
typology  of  specific  Hillwood  Junior  High  characteristics. 

Comparison  of  Maple  Grove  Middle  School  and  Hillwood  Junior  High  School 

The  typologies  associated  with  Maple  Grove  Middle  School  and  Hillwood  Junior  High  are  evident  and  can 
be  seen  in  a variety  of  ways.  Maple  Grove  is  an  affluent  organization  with  a wide  variety  of  instructional  technology 
available  for  members  to  draw  upon  while  Hillwood  continues  to  work  diligently  and  carefully  to  create  a playing 
field  that  will  provide  more  and  more  opportunities  for  its  teachers.  At  Maple  Grove,  players  enjoy  freedom  to  make 
decisions  about  day-to-day  classroom  activities  without  pressures  from  the  building  principal  or  district 
administration;  conversely,  Hillwood  principals  exert  strong  influence  over  the  members  of  the  group  on  pedagogy 
and  curriculum.  There  are  few  rules  at  Maple  Grove  where  Hillwood  teachers  have  distinct  rules  and  limitations  on 
choices.  Maple  Grove  is  an  instructional  technology-rich  environment  and  those  tools  are  used  in  a wide  variety  of 
ways,  whereas  Hillwood  has  limited  instructional  technology  tools  and  many  of  the  uses  of  those  tools  are  regulated 
by  the  administration. 

The  grid  and  group  typology  was  useful  in  describing  the  organizational  culture  in  both  of  the  school  sites, 
and  the  typology  was  broad  enough  to  encompass  the  variety  of  social  interactions  and  contexts  surrounding 
instructional  technology  use. 
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Summary  of  the  Study 

Schools  like  Maple  Grove  and  Hillwood  have  increased  the  amount  of  technology  available  to  teachers, 
assisted  in  increasing  teachers’  familiarity  with  technology  through  training,  and  supported  and  encouraged  the  use 
of  technology  through  administrative  directives.  In  spite  of  these  efforts  to  increase  the  use  of  instructional 
technology,  some  teachers  are  not  using  instructional  technology  or  use  them  only  at  minimal  levels  as  prescribed  by 
administrators. 

Why  do  a small  number  of  teachers  embrace  change  and  demonstrate  high-level  competence  and 
integration  of  instructional  technology  while  other  teachers  adamantly  refuse  to  adopt  or  even  experiment  with 
technology? 

In  an  attempt  to  answer  this  question,  this  study  employed  the  lens  of  Mary  Douglas’  (1982)  grid  and  group 
typology  as  a framework  and  vocabulary  to  examine  teachers’  use  of  instructional  technology  in  two  Midwestern 
school  sites.  Specifically,  the  study  looked  at: 

1.  How  instructional  technology  is  used  in  classrooms  in  each  of  the  selected  schools; 

2.  Ways  in  which  the  use  of  instructional  technology  reflect  grid/group  realities  in  each  of  the 
selected  schools; 

3.  Other  realities  as  revealed  in  each  of  the  selected  schools;  and 

4.  Whether  grid  and  group  typology  was  helpful  in  understanding  differences  in  teachers' 
instructional  technology  use  in  the  selected  schools. 

The  schools  selected  for  the  case  study  and  analysis  were  a middle  school  and  a junior  high  school  in  rural 
towns  located  in  a Midwestern  state.  The  schools  were  selected  based  on  Stake’s  (1995)  assertion  that  the  sites  be 
easily  accessible  by  the  researcher  and  be  welcoming  of  the  intrusion  of  the  researcher  into  the  site.  Additionally,  the 
two  schools  were  chosen  through  purposive  sampling  (Stake,  1995)  because  both  exhibited  some  use  of  instructional 
technology  by  teachers  in  the  schools  and  were  different  in  demographics,  financial  strength,  and  technology 
emphasis  across  the  school  site. 

Data  collection  occurred  using  a variety  of  methods  including  interviews,  observations,  document  analysis, 
and  a brief  questionnaire.  The  purpose  of  the  data  was  to  provide  for  a description  of  teacher  use  of  instructional 
technology  within  the  school  site  and  to  understand  the  school’s  organizational  culture  within  which  that 
instructional  technology  use  occurred. 

Data  collection  occurred  over  a period  of  months  beginning  in  October  2003  and  ending  in  March  2004; 
data  analysis  was  ongoing  throughout  the  study.  Through  multiple  interviews,  documents,  observation,  discussion 
with  academic  advisors  and  peers,  member  checks  from  participants  in  the  study,  and  the  use  of  a forced-choice  pair 
survey,  triangulation  of  the  data  was  realized. 

Summary  of  the  Findings 

Maple  Grove,  an  Individualist  (weak  grid,  weak  group)  culture,  is  a school  where  teachers  enjoy  a high 
degree  of  autonomy  and  the  freedom  to  make  decisions  about  their  classrooms  as  they  see  fit.  Teachers  have  access 
to  nearly  any  piece  of  instructional  technology  they  need  and  are  able  to  negotiate  acquisition  for  those  items  they  do 
not  have.  A modified  schedule  previously  allowed  teachers  the  opportunity  to  have  extended  time  on  Friday 
afternoons  each  week  for  professional  development  usually  centered  around  instructional  technology  integration  or 
training  on  new  technology  acquired  by  the  school;  however  during  the  current  school  year,  that  practice  has  been 
abandoned  for  a more  traditional  school  schedule.  Finally,  a recent  change  in  superintendents  has  caused  a shift  in 
focus  that  does  not  emphasize  the  use  of  instructional  technology  that  has  pleased  some  members  of  the  faculty  and 
disappointed  others. 

Hillwood  Junior  High  School’s  teachers  operate  in  a Corporate  (strong  grid,  strong  group)  culture  where 
the  school  administration  has  established  a set  of  rules  and  roles  to  be  followed  in  the  classroom.  Decisions  about 
changes  in  classroom  teaching  strategies  are  approved  by  the  administration.  Teachers  at  Hillwood  are  willing  to 
forgo  their  personal  needs  for  the  needs  of  the  school:  they  cover  each  other’s  extended  absences,  they 
collaboratively  raise  funds  for  classroom  materials,  and  they  serve  as  peer  resources  for  others  in  the  school  setting. 
They  do  not  have  ready  access  to  instructional  technology  and  do  not  enjoy  an  ability  to  negotiate  easily  for 
acquisition  of  new  resources.  Decisions  about  instructional  technology  purchases  are  made  at  the  administrative 
level  and  administrators  do  not  often  ask  for  input  from  teachers  on  purchasing  decisions. 

Conclusions 

The  following  research  questions  guided  this  study  and  are  discussed  below. 

How  is  instructional  technology  used  in  classrooms  in  each  of  the  selected  schools?  Maple  Grove  is  a 
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technology -rich  environment  that  encourages  teachers  to  make  decisions  about  instructional  technology  as  best  they 
see  fit  for  their  individual  classes.  Administrators  support  the  use  of  technology  at  Maple  Grove  and  support 
teachers’  efforts  to  integrate  instructional  technology  into  their  teaching. 

Uses  of  instructional  technology  at  Maple  Grove  ranged  from  simple  document  word  processing  and 
information  retrieval  from  the  World  Wide  Web  via  the  Internet  to  more  advanced  uses  of  applications  such  as 
Microsoft  PowerPoint,  Excel,  and  Access.  There  is  evidence  of  students  producing  HTML -based  content  for  the 
World  Wide  Web,  and  uses  of  digital  still  photography  and  digital  video  production.  Teachers’  use  of  instructional 
technology  also  ranged  across  a wide  variety  of  application  including  all  those  previously  mentioned.  In  addition, 
teachers  use  PowerSchool  as  a student  information  management  system  for  reporting  grades  and  attendance.  Parents 
and  students  can  access  PowerSchool  to  keep  up-to-date  on  student  progress.  Teachers  also  provide  web-based 
information  sources  to  keep  parents  and  students  informed  of  classroom  events,  assignments,  and  other  school 
happenings. 

Teachers  and  administrators  at  Maple  Grove  also  spoke  of  instructional  technology  as  being  an  important 
part  of  the  curriculum  and  a part  of  the  school’s  mission,  and  they  discussed  how  instructional  technology  is  part  of 
the  world  outside  of  Maple  Grove  that  students  are  being  prepared  to  enter.  Mr.  Randalia,  the  principal,  hoped  that 
technology  might  provide  some  motivation  to  a group  of  students  who  have  little  in  their  personal  lives  to  motivate 
them  and  few  role  models  to  push  them  to  greater  achievements  outside  of  Maple  Grove.  In  this  way,  technology  is 
a liberating  tool  that  serves  to  give  students  access  to  a world  beyond  Maple  Grove. 

Maple  Grove  may  take  on  this  mission  because  individuals  in  the  district  in  the  past  have  envisioned  that 
technology  might  serve  this  role.  Mr.  Fayette,  the  former  superintendent,  put  in  place  the  infrastructure  to  support 
this  mission  and  laid  the  foundation  for  maintaining  the  infrastructure.  For  many  years  to  come.  Maple  Grove  will 
not  need  to  plan  for  funding  the  infrastructure;  but  they  will  need  to  envision  ways  to  encourage  additional  use  of 
instructional  technology.  However,  that  direction  will  need  to  be  in  keeping  with  the  Individualist  organizational 
culture  already  in  place  at  Maple  Grove.  If  there  is  not  an  effort  to  provide  continuing  training  and  motivation  to 
teachers  at  Maple  Grove  in  the  areas  of  instructional  technology  integration,  there  will  be  less  and  less  use  of 
technology. 

Hillwood  Junior  High  School  presents  a technology  environment  that  is  much  less  advanced  than  the 
environment  evidenced  at  Maple  Grove.  Teachers  have  a single  computer  in  their  classroom,  but  many  of  them  are 
older  model  computers  with  slow  processors  and  operating  systems  as  old  as  Windows  95  and  Windows  98. 

Teachers  at  Hillwood  are  at  times  envious  of  other  schools  that  have  better  instructional  technology  tools  and  are 
sometimes  frustrated  with  the  lack  of  tools  available  to  them. 

Instructional  technology  use  at  Hillwood  is  limited.  Teachers  are  required  to  submit  grades  electronically 
and  email  hourly  attendance  records  to  the  school  secretary.  They  also  use  instructional  technology  to  prepare 
curriculum  materials  such  as  worksheets,  handouts,  quizzes,  and  tests.  On  occasion,  they  may  schedule  time  in  the 
computer  lab  for  student  word  processing  or  Internet  searching  via  the  World  Wide  Web.  But  for  many  teachers, 
‘computers’  is  another  subject  in  the  curriculum  taught  by  the  ‘computer  teacher’,  and  they  do  not  see  ways  in  which 
they  might  integrate  into  their  teaching  the  instructional  technology  already  in  place  in  the  school. 

Student  use  of  technology  is  severely  limited  to  the  point  that  s ome  teacher  computers  are  physically 
labeled  “Teacher  Use  Only!’’  For  many,  the  use  of  a computer  during  the  school  day  is  a reward  for  good  behavior. 
Occasional  lab  use  is  evident,  but  even  that  is  seen  as  a reward  as  revocation  of  the  privilege  is  threatened  as  a 
consequence  of  bad  behavior. 

Teachers  have  indicated  that  one  of  the  reasons  for  a lack  of  integration  of  instructional  technology  is  the 
lack  of  access  to  hardware.  Hillwood  administration  is  focusing  attention  on  this  aspect  by  hiring  an  outside 
consulting  firm  to  evaluate  technology  in  the  district  and  assist  in  creating  a plan  for  reshaping  the  instructional 
technology  landscape  in  the  district.  It  will  be  important  for  Hillwood’ s success  that  in  addition  to  investing  in 
hardware  and  software,  they  also  invest  in  training  and  mentoring  for  teachers.  This  training  and  mentoring  will 
need  to  be  rather  basic  at  the  outset,  beginning  with  fundamental  operations,  as  new  technology  added  to  the  district 
will  be  unlike  existent  technology.  However,  the  researcher  thinks  there  is  an  environment  in  place  at  Hillwood  that 
will  encourage  the  adoption  and  integration  of  instructional  technology.  As  a strong  group  culture,  Hillwood  has 
demonstrated  commitment  to  mission  of  the  group  and  already  has  in  place  the  support  structures  needed  for  more 
advanced  instructional  technology  users  to  mentor  those  less  proficient.  Once  Hillwood  makes  the  planned 
substantial  investment  in  technology  infrastructure,  they  must  draw  upon  the  strong  group  culture  to  make  best  use 
of  the  technology. 

In  what  ways  does  the  use  of  instructional  technology  reflect  grid/group  realities  in  each  of  the  selected 
schools?  Maple  Grove’s  Individualist  (weak  grid,  weak  group)  culture  is  evidenced  by  its  instructional  technology 
use.  In  an  Individualist  culture,  individuals  are  afforded  much  leeway  in  making  decision  and  there  is  less  emphasis 


775 


on  the  contributions  to  or  maintenance  of  the  group  as  a whole. 

Teachers  at  Maple  Grove  are  allowed  to  make  decisions  about  instructional  technology  integration  without 
pressure  from  district-  or  building-level  administrators.  In  only  one  way  are  teachers  expected  to  use  technology  in 
their  work:  the  required  use  of  PowerSchool  as  a student  information  management  system.  Weak  grid  indicates 
much  individual  choice,  and  teachers  evidence  this.  Some  teachers  make  extensive  use  of  instructional  technology  in 
their  classes  even  to  the  point  that  they  admit  having  a difficult  time  envisioning  themselves  teaching  without 
instructional  technology  or  even  with  reduced  access.  Other  teachers  use  instructional  technology  only  to  the  levels 
required  by  the  administration.  In  addition,  administrators  encourage  instructional  technology  use  by  providing  the 
necessary  tools  and  allow  teachers  to  negotiate  individually  for  additional  hardware  and  software  as  these  needs 
become  apparent  to  each  individual. 

Throughout  Maple  Grove  Middle  School,  other  evidence  of  individuation  is  apparent.  Students  are  given 
access  to  extensive  hardware  and  software  tools  on  an  as  needed  basis.  They  are  allowed  to  take  computers  home  if 
a need  is  shown  and  are  allowed  to  use  building  computing  facilities  unsupervised.  Teachers  at  Maple  Grove  take  on 
a variety  of  roles,  both  ascribed  and  assumed.  Some  teachers  act  as  peer  mentors  to  other  teachers  in  the  process  of 
integrating  instructional  technology,  and  technology  support  personnel  roles  have  changed  over  time  as  needs  of  the 
school  have  evolved.  Teachers  also  realize  a relaxed  emphasis  on  instructional  technology  at  Maple  Grove  with  the 
change  in  district  administration. 

Maple  Grove  Middle  School’s  teachers’  use  of  instructional  technology  also  demonstrates  the  weak  group 
aspect  of  an  Individualist  culture.  In  such  a culture,  members  of  the  group  do  not  place  emphasis  on  the  continuation 
of  the  group,  nor  is  there  emphasis  on  the  creation  of  relationships  within  the  group.  Members  do  not  draw  their 
identity  from  the  group  and  are  able  to  disconnect  from  the  group  outside  the  group’s  immediate  environs.  The 
group  is  not  withdrawn  from  outsiders.  There  are  few  collaborative  efforts  underway  at  Maple  Grove.  Teachers  do 
not  work  in  teams  and  do  not  plan  lessons  in  concert  with  one  another.  Staff  members  live  outside  the  district  and  do 
not  interact  socially  outside  the  school  day.  The  school  is  open  to  outsiders  and  has  been  since  it  has  emerged  as  a 
leader  in  instructional  technology  use.  In  fact,  former  employees  spoke  of  the  “daily  tour’’  by  outsiders  of  Maple 
Grove’s  facilities. 

There  is  less  emphasis  on  creating  an  atmosphere  of  instructional  technology  integration  at  Maple  Grove. 

In  the  past,  the  school  district  adopted  a modified  schedule  that  allowed  students  to  be  dismissed  at  midday  on 
Friday  and  provided  a block  of  time  for  teacher  training  in  new  technologies  and  planning  for  instructional 
technology  integration.  Technology  support  personnel  cite  this  loss  of  dedicated  instructional  technology  training 
and  planning  time  as  a major  contributing  factor  in  the  deemphasizing  of  instructional  technology  use  at  Maple 
Grove. 

Finally,  there  is  no  sense  at  Maple  Grove  among  administrators  or  teachers  that  the  deemphasizing  of 
instructional  technology  integration  is  problematic.  While  some  teachers  and  technology  support  personnel 
bemoaned  the  loss  of  Friday  afternoons  and  the  block  of  time  it  afforded  them,  no  one  spoke  of  disappointment  that 
facilities  went  unused  or  that  some  teachers  did  not  use  instructional  technology  as  extensively  as  other  teachers  did. 

Maple  Grove’s  use  of  instructional  technology  is  evidence  of  its  Individualist  cultures:  much  flexibility  in 
how  individual’s  acquire  and  use  instructional  technology  and  little  emphasis  on  creating  a group  culture  of 
instructional  technology -using  teachers. 

Hillwood  Junior  High  School’s  Corporate  (strong  grid,  strong  group)  culture  is  also  evidenced  by  the  use  of 
instructional  technology,  but  not  as  clearly  as  Maple  Grove.  In  a Corporate  environment,  there  is  a distinct  set  of 
rules  by  which  individuals  must  operate  and  each  individual  plays  a distinct  role  in  the  organization.  Additionally, 
there  is  a strong  sense  of  group  among  the  members  to  the  point  where  members  work  at  creating  sense  of  group 
identity. 

Hillwood  teachers  are  limited  in  the  choices  they  may  make  about  instructional  technology  integration  in 
that  there  are  not  extensive  resources  available  to  them,  and  the  building  principal  exerts  strong  influence  over 
curricular  matters.  While  the  school  has  set  a goal  of  placing  on  each  teacher’s  desk  a computer  and  printer,  those 
existent  computers  are  out-dated  in  both  computing  power  and  operating  system.  There  exists  a single  computer 
laboratory  that  is  prioritized  to  the  technology  education  teacher  and  other  teachers  are  allowed  entry  only  when  it 
does  not  conflict  with  that  schedule.  In  addition,  there  is  a mobile  computer  lab  consisting  of  14  laptop  computers, 
but  that  resource  does  not  see  much  use  by  classroom  teachers.  Future  plans  indicate  the  addition  of  computer 
laboratory  facilities  beyond  the  current  facilities,  yet  access  will  be  allocated  on  a weekly  basis,  and  there  are  plans 
to  install  a pre -determined  curriculum  on  the  computers.  Such  allocation  of  resources  and  decision-making  on  the 
part  of  administrators  limits  the  ways  in  which  teachers  may  integrate  instructional  technology.  Teachers  are 
required  by  the  administration  to  utilize  technology  in  reporting  grades  and  reporting  attendance. 

Evidence  points  to  a need  to  discuss  with  the  building  principal  any  planned  innovations  in  curriculum  even 
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to  the  point  that  classroom  volunteers  should  be  cleared  by  the  building  principal  prior  to  their  service  in  the 
classroom.  Teachers  have  been  directed  in  which  ways  they  may  integrate  instructional  technology  in  their  classes 
and  have  also  been  directed  in  which  ways  they  may  not:  mathematics  teachers  may  not  use  calculators  in  their 
teaching.  The  building  principal  cites  as  his  reason  that  students  are  not  allowed  to  use  calculators  on  standardized 
tests  used  by  the  school,  and  he  does  not  wish  them  to  become  dependent  upon  them  for  doing  mathematics 
problems. 

Teachers  also  have  little  choice  in  negotiating  for  additional  resources.  Acquisition  of  resources  is 
controlled  by  the  administrative  team  consisting  of  the  district  superintendent  and  building  principals.  This  team 
accepts  purchase  requisitions  and  generally  holds  them  until  the  start  of  the  next  budget  year  in  order  to  be  certain 
that  requests  meet  the  needs  of  the  maximum  number  of  students.  Even  though  teachers  may  feel  a need  for  a 
particular  purchase,  they  are  limited  only  to  requesting  the  resource  and  then  awaiting  administrative  approval  that 
may  be  delayed  until  the  start  of  the  next  year. 

There  is  evidence  of  the  strong  group  aspect  of  a Corporate  culture  at  Hillwood  as  well.  In  a strong  group 
culture,  there  is  evidence  of  group  members  dedicated  to  the  mission  of  the  group.  Members  of  the  group  develop 
relationships  outside  the  immediate  environs  of  the  group,  relationships  that  extend  to  their  personal  lives  as  well. 

Teachers  at  Hillwood  live  in  the  district  and  actively  participate  in  school  life  beyond  the  classroom.  They 
have  taken  on  each  other’s  duties  in  the  event  of  long-term  illness,  and  they  attend  school  events  as  social  outings. 
The  school  office  is  both  a professional  and  a social  gathering  place.  They  depend  upon  each  other  for  assistance 
with  instructional  technology  integration  and  band  together  to  attend  training  off-site  to  maximize  the  return  on  such 
training,  especially  if  attendance  means  additional  resources  be  given  to  the  school.  Teachers  even  give  of  their 
personal  time  in  order  to  raise  funds  to  purchase  resources  that  might  not  otherwise  be  available  through  traditional 
means.  A strong  parent  organization  has  donated  funds  for  purchases  as  well.  Prior  to  moving  to  their  current 
physical  space,  teachers  at  Hillwood  were  required  to  share  classrooms  and  classroom  resources. 

What  other  realities  were  revealed  in  each  of  the  selected  schools?  Leadership  is  an  integral  part  of  any 
organization  and  especially  important  in  determining  an  organization’s  use  of  instructional  technology.  In  each  of 
the  two  typologies  discovered  in  this  study,  leadership  styles  are  different.  Both  Maple  Grove  Middle  School  and 
Hillwood  Junior  High  School  have  recently  undergone  dramatic  leadership  changes  in  some  fashion.  Maple  Grove’s 
lasses-faire  leadership  by  the  building  administration  is  fundamental  in  its  Individualist  (strong  grid,  strong  group) 
organizational  culture.  Mr.  Randalia,  the  principal,  allows  teachers  to  operate  their  classrooms  as  they  see  fit,  using 
instructional  technology  in  ways  with  which  they  are  most  comfortable,  with  the  exception  of  the  requirement  to  use 
technology  for  grade  and  attendance  reporting.  For  him  to  begin  to  issue  directives  for  instructional  technology  use 
would  not  complement  the  existing  organizational  culture.  Maple  Grove  is  also  adjusting  to  a new  district 
administration  with  a shifting  emphasis  in  the  role  instructional  technology  plays.  The  former  superintendent  was 
emphatic  in  his  instructional  technology  integration  expectations  even  including  technology  use  as  part  of  the 
standard  teacher  evaluation.  While  the  technology  portion  of  the  evaluation  was  removed  prior  to  the  current 
superintendent’s  arrival,  she  nonetheless  has  indicated  to  teachers  that  she  will  not  be  emphasizing  the  use  of 
instructional  technology  as  her  predecessor. 

Hillwood  is  coming  to  terms  with  a change  in  physical  plant  facilities  and  in  a restructuring  of 
administration  at  the  building  level.  Both  of  these  situations  present  issues  of  leadership:  a reorganized  physical 
plant  means  that  teachers  each  have  their  own  classroom  and  no  longer  need  share  resources,  but  it  also  means  that 
additional  resources  must  be  acquired  to  meet  the  goal  of  placing  a computer  on  every  teacher’s  desk.  Teachers  in 
the  current  junior  high  had  worked  under  the  direct  supervision  of  Mr.  Randall  Washington  in  the  former 
intermediate  school  configuration.  The  separation  of  the  school  into  two  entities,  an  intermediate  school  of  grades  4- 
6 and  a junior  high  of  grades  7-9,  has  placed  junior  high  teachers  under  the  direct  supervision  of  Mr.  Charles 
Riverside,  though  Mr.  Washington  is  technically  still  the  principal  of  record  of  Hillwood  Junior  High,  and  he  and 
Mr.  Riverside  work  in  concert  to  make  decisions  that  they  believe  are  best  for  Hillwood’ s students. 

There  are  teachers  at  Maple  Grove  who  have  access  to  extensive  instructional  technology  resources  and 
make  little  use  of  those  resources.  Likewise,  there  are  teachers  who  do  make  use  of  the  instructional  technology 
available  to  them.  Conversely,  at  Hillwood,  there  is  evidence  of  teachers  with  few  instructional  technology  resources 
who  work  diligently  to  use  what  few  tools  are  available  to  them  as  they  feel  it  is  their  duty  to  do  so,  and  there  are 
teachers  who  are  content  to  use  instructional  technology  only  at  the  levels  required  by  the  administration. 

Teachers  at  both  Maple  Grove  and  Hillwood  are  neither  rewarded  nor  punished  for  their  use  or  non-use  of 
instructional  technology.  The  motivation  to  use  instructional  technology  and  the  rewards  that  accompany  its  use  are 
intrinsic.  Teachers  use  instructional  technology  because  they  believe  they  should  and  they  believe  there  are  benefits 
attached  to  doing  so;  others  choose  not  to  use  instructional  technology.  In  either  case,  the  administration  neither 
encourages  nor  discourages  instructional  technology  use. 
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Was  grid/group  helpful  in  understanding  differences  in  teachers’  instructional  technology  use  in  the 
selected  schools?  Douglas  (1989)  described  the  use  of  the  grid/group  typology:  “The  most  interesting  questions 
[grid/group]  is  designed  to  answer  are  about  attitudes,  values,  and  established  thought  patterns  which  correlate  with 
particular  grid/group  positions”  (p.  175).  Douglas’s  typology  provided  a vocabulary  with  which  to  discuss  the 
organizational  culture  of  each  school  and  to  describe  instructional  technology  use  within  that  culture.  Information 
from  interviews,  observations,  and  document  analysis  contributed  to  an  understanding  of  teachers’  attitudes,  values, 
and  established  thought  patterns,  and  data  helped  establish  a position  within  the  typology  for  each  school.  However, 
it  is  critical  to  note  that  the  typology  is  not  one  of  absolutes;  rather,  each  axis  of  the  typology  serve  as  a continuum 
along  which  a determination  is  made  by  the  researcher  as  to  the  strength  or  weakness  of  the  organization  under 
study.  In  addition,  since  the  combination  of  two  axes  represents  a wide  range  of  possibilities,  it  was  difficult  at  times 
to  place  each  school  definitively  on  each  axis.  Certain  aspects  discovered  at  each  school  indicated  both  strong  and 
weak  grid  and  group  elements;  a determination  had  to  be  made  as  to  which  outweighed  the  other. 

The  availability  of  a vocabulary  with  which  to  describe  both  the  organizational  culture  of  each  school  and 
instructional  technology  use  among  its  members  was  useful.  Douglas’s  typology  allowed  for  discussion  of  both 
individual  choice  and  the  relationships  created  by  individuals.  In  addition,  the  typology  allowed  for  a differentiation 
between  Maple  Grove’s  wide  range  of  choices  and  few  restrictions  placed  on  them  by  administration  and 
Hillwood’s  narrow  set  of  choices  restricted  by  the  limited  instructional  technology  available  to  them  and  the  strong 
emphasis  on  being  a part  of  an  established  group  evident  at  Hillwood. 

The  typology  was  useful  in  helping  to  understand  differences  in  instructional  technology  use  by  teachers  at 
Maple  Grove  Middle  School  and  Hillwood  Junior  High  School. 


Figure  1 

Mary  Douglas’  Typology  of  Social  Environment  Prototypes 
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Figure  2 

Maple  Grove  Middle  School’s  Grid/Group  Typology 
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Figure  3 

Hillwood  Junior  High  School’s  Grid/Group  Typology 
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Abstract 

This  study  focuses  on  the  effects  of  map  display  and  modality  on  learning  geographical  maps  in  a 
computer-based  environment.  Participants  were  randomly  assigned  to  four  versions  of  a computer  program  created 
by  crossing  two  levels  of  map  display  ( hypertext  vs.  rollover)  with  two  levels  of  modality  (audio  vs.  audio  and  text). 
Results  showed  that  the  modality  factor  affects  recall  and  inference  test  performance  of  college  students  in  map 
learning  significantly. 


Purpose 

Technology  has  improved  so  much  today,  that  it  is  easy  to  teach  in  ways  that  are  both  interactive  and 
communicative.  The  Internet  and  the  WWW  have  added  a new  dimension  to  teaching  and  learning.  It  not  only 
integrates  different  mediums  but  also  their  design,  development  and  implementation.  With  the  paradigm  shift  from 
teaching  to  learning,  mu  ltimodality  offers  an  excellent  way  for  self-paced  and  exploratory  learning  more  in  tune 
with  the  constructivist  methodology  than  the  behaviorist  methodology.  According  to  Mayer  (1997),  meaningful 
learning  “occurs  when  learners  select  relevant  information  from  what  is  presented,  organize  the  pieces  of 
information  into  a coherent  mental  representation,  and  integrate  the  newly  constructed  representation  with  others”. 

Theoretical  Framework 

Results  from  over  forty  studies  have  shown  that  geographic  maps  can  be  used  as  cognitive  tools  to  increase 
the  recall  of  related  instructional  text.  The  image  of  a map  is  an  excellent  mnemonic  device  to  associate  geographic 
locations  contained  in  the  map  with  related  facts  in  the  text  (Kulhavy,  Stock,  & Kealy,  1993).  Maps  integrate  both 
feature  information  and  structure  information  which  create  an  integrated  image  of  the  display  (Verdi,  Johnson, 
Stock,  Kulhavy,  and  Ahern,  1997).  This  integration  results  in  an  economical  unit  that  makes  it  easy  for  learners  to 
switch  their  attention  across  an  image  while  simultaneously  processing  information  from  a related  text  (Larkin  and 
Simon,  1987).  The  feature -to-fact  referential  connections  (Mayer  and  Anderson,  1992)  create  additional  retrieval 
cues  that  learners  can  use  to  improve  their  recall  of  map  and  text  information  (Verdi,  et  ah,  1997). 

Paivio's  dual  coding  theory  (Paivio,  1986)  contends  that  the  cognitive  system  functions  with  two  symbolic 
systems  that  are  distinct,  yet  referentially  connected.  The  verbal  system  stores  information  generated  through  the 
processing  of  language  while  the  nonverbal  system  stores  information  received  during  the  processing  of  images. 
Conjoint  retention  (CR)  theory,  an  extension  of  dual  coding  theory  (Paivio,  1986)  has  been  applied  to  map  learning. 
According  to  CR  theory,  maps  are  stored  spatially  in  the  nonverbal  system  as  intact  units  while  text  information  is 
stored  linearly  in  the  verbal  system.  The  presence  of  referential  connections  between  these  two  separate  memory 
stores  can  help  in  the  study  of  geographical  maps. 

Computer  environments  offer  a far  greater  scope  to  make  use  of  dual  coding  theory  than  print  media.  It 
offers  far  greater  scope  to  use  multimedia.  Print  maps  are  usually  drawn  on  one  page  and  the  opposite  page  has  the 
narrative.  Sorting  through  the  narrative  to  reference  a particular  location  is  not  only  time-consuming  but  also  leads 
to  extrinsic  cognitive  load.  Extrinsic  load  can  be  brought  down  to  a large  extent  by  bringing  in  the  contiguity  factor. 
According  to  the  contiguity  principle,  instructions  in  multimedia  are  more  effective  when  words,  pictures  and  audio 
are  presented  contiguously,  rather  than  separately  in  time  or  space.  The  contiguity  factor  takes  into  account  the 
severe  limitations  of  the  human  processing  system  (Baddeley,  1986)  which  leads  to  cognitive  load.  In  a computer 
environment  it  is  possible  to  have  feature -to -fact  referential  connections  both  in  terms  of  temporal  contiguity  and 
spatial  contiguity  as  well  as  an  image  of  the  map  with  or  without  an  accompanying  text  and  audio.  Contiguity  helps 
to  bring  cross -code  referential  links  lowering  the  cognitive  load  and  thus  helping  in  better  cognition  of  the 


782 


geographical  content.  According  to  Mayer  (2002)  spatial  contiguity  occurs  when  instructional  elements  such  as  text 
and  image  or  text  and  audio  or  image  and  audio  are  physically  close.  In  a computer  environment  this  occurs  when 
the  instructional  elements  are  on  the  same  screen  rather  than  on  different  screens.  This  is  achieved  by  having  a roll- 
over effect.  When  the  cursor  is  over  an  image  or  at  a particular  location  the  text  immediately  pops  up  or  the  audio 
message  immediately  starts.  But  since  in  this  situation  the  text  and/  or  the  audio  message  pop  up  immediately  on  the 
same  screen  this  could  very  well  be  considered  to  have  both  the  temporal  and  spatial  contiguity  simultaneously. 
However  in  the  hypertext  effect  the  learner  selects  a map  feature  by  clicking  on  it  with  a computer  mouse.  This 
opens  a new  screen  with  the  text  and  or  audio  in  it.  Since  separate  screens  are  involved  here  the  instructional 
elements  are  separated  in  time  and  not  space,  it  can  be  considered  to  have  temporal  contiguity. 

This  study  examines  the  effects  of  text  and  audio  (dual  modality)  and  audio  only  in  learning  from  an  online 
map  in  temporal  with  spatial  contiguity  conditions  and  temporal  only  conditions.  The  question  that  is  being  studied 
is  to  what  extent  modality  plays  a role  in  learning  from  an  online  map.  The  second  question  being  looked  into  is 
under  what  contiguity  conditions  modalities  affect  learning.  It  is  hypothesized  that  dual  modality  will  influence 
learning  under  both  contiguity  conditions  because  the  feature  to  text  cross-reference  is  being  further  augmented  by 
using  dual  channels  of  visual  and  auditory  senses.  This  augmentation  should  help  learners  to  make  better  use  of  their 
information  processing  system. 

Design  and  Subject 

One  hundred  and  seventy-six  undergraduate  students  from  a large  southwestern  university  were  randomly 
assigned  to  one  of  four  conditions  in  a 2 X 2 factorial  design.  There  were  two  levels  of  map  display  (hypertext  vs. 
rollover)  which  were  crossed  with  two  levels  of  modality  (audio  vs.  audio  and  text)  to  form  the  factorial  design. 
Students  were  given  a fictitious  map  with  twenty  locations  having  fictitious  names  on  it.  Each  location  had  a fact 
describing  its  unique  feature.  Depending  on  the  condition  they  were  assigned  to,  students  would  either  receive  an 
audio  message  (audio  modality)  or  both  audio  and  text  message  (dual  modality)  telling  them  the  unique  feature  of 
the  location.  Students  could  access  this  information  by  either  clicking  on  the  location  (hypertext)  using  the  computer 
mouse  when  the  embedded  text  with  the  narration  or  the  audio  alone  would  open  up  in  a different  screen  or  by 
placing  the  cursor  on  a particular  location  using  the  computer  mouse  when  the  text  with  the  narration  or  the  audio 
alone  would  be  presented  contiguously  with  the  location.  Students  received  course  credit  for  participation. 

Materials 

The  materials  consisted  of  a computer-based  reference  map  depicting  fictitious  feature  names  from  the 
island  of  Malta  and  two  manipulations  of  a 470-word  text  (i.e.,  hypertext  and  rollover)  containing  facts  about  the 
features.  Text  narration  with  audio  could  be  heard  through  headphones. 

The  map  contained  20  fictitious  features  randomly  distributed  across  the  map  surface.  There  was  one  fact 
associated  with  each  map  feature. 


Procedure 

All  participants  were  given  verbal  instructions,  guiding  them  to  learn  as  much  as  possible  from  the  map.  All 
participants  were  given  20  minutes  to  study  the  map.  After  studying  the  map,  all  participants  were  given  three  paper- 
based  instruments  with  two  retrieval  measures  and  one  inference  measure,  to  determine  the  effects  of  the  treatment 
conditions.  A free  recall  measure  was  used  to  test  participants  on  the  map  features  and  facts  they  studied  for  which 
they  were  given  10  minutes.  Participants  were  then  given  eight  minutes  to  take  the  second  retrieval  measure  which 
was  a map  reconstruction  task.  They  were  given  a sheet  with  the  computer  screen  border  drawn  for  this  task.  Finally, 
a twelve-item  multiple -choice  test  designed  as  the  inference  measure  was  given  to  assess  the  participants'  ability  to 
make  inferences  about  the  information  on  the  map.  To  correctly  answer  the  items  the  participants  had  to  infer 
information  from  the  map  that  was  not  specifically  stated  in  the  materials.  This  was  viewed  as  a higher-order 
thinking  task  as  opposed  to  free  recall.  This  measure  was  scored  by  giving  one  point  for  each  item  answered 
correctly.  All  statistical  tests  were  performed  with  an  alpha  of  .05. 

Results  and  Conclusions 

ANOVA  revealed  a main  effect  on  the  modality  variable  for  fact  recall  F(l,  73)  = 6.008,  p = .017.  Further 
analysis  revealed  significant  display  type  by  modality  interaction  effects  on  the  inference  test  F(l,  73)  = 5.616, 
p=.020  and  the  name  recall  test  F(  1 ,73)  =4.384,  p=.040.  Analysis  of  the  inference  test  interaction  shows  that 
hypertext  (temporal  contiguity  only)  students  in  the  text  and  audio  conditions  (dual  modality)  outperformed  students 
in  the  hypertext  (temporal  contiguity  only)  in  the  audio  only  conditions,  but  the  opposite  effect  occurred  in  the 
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rollover  conditions.  Further,  analysis  of  the  name  recall  interaction  revealed  that  roll-over  (temporal  and  spatial 
contiguity)  students  in  the  audio  only  condition  outperformed  students  in  the  roll-over  (temporal  and  spatial 
contiguity)  text  and  audio  (dual  modality)  condition  for  the  name  recall,  while  the  opposite  effect  occurred  for  those 
in  the  hypertext  conditions. 

The  results  are  partly  in  tune  with  the  hypothesis.  Dual  modality  does  have  a positive  effect  on  learners  but 
not  in  both  the  contiguity  conditions.  It  appears  to  help  learners  more  in  a temporal  contiguity  condition  than  those 
in  temporal  and  spatial  contiguity,  which  can  be  considered  as  dual  contiguity.  Dual  modality  with  dual  contiguity 
does  not  seem  to  serve  its  purpose.  Dual  contiguity  with  single  modality  (audio  in  this  case)  and  single  contiguity 
(temporal  in  this  case)  with  dual  modality  seems  to  do  a better  job  at  improving  understanding  of  geographical 
maps.  This  could  be  explained  in  terms  of  the  cognitive-load  theory.  Two  simultaneous  contiguity  conditions 
(rollover)  with  both  audio  and  visual  gives  rise  to  unwanted  redundancy.  This  in  turn  may  generate  a heavy 
cognitive  load  that  is  unfavorable  to  learning  (Chandler  & Sweller,  1991;  Sweller,  1988,  1989;  Sweller,  Chandler, 
Tierney,  & Cooper,  1990).  It  is  also  possible  that  the  speed  with  which  learners  read  the  text  could  be  different  from 
the  speed  of  the  audio  which  might  not  have  been  conducive  to  learning.  In  the  single  audio  modality  this 
overlapping  is  absent  and  learners  have  performed  better.  Another  possibility  could  be  that  learners  in  the  rollover 
condition  have  a split -attention  problem  as  they  have  to  use  the  visual  channel  for  both  the  image  of  the  map  as  well 
as  to  read  the  text.  Addition  of  audio  to  this  scenario  has  not  helped. 

The  hypertext  condition  does  better  in  dual  modality  as  there  is  no  split  attention  problem.  On  clicking  on  a 
location  the  text  opens  up  in  a new  screen.  Since  the  image  of  the  map  is  not  visible  anymore,  learners  are  able  to 
concentrate  on  the  text  and  audio.  But  when  audio  alone  is  presented  learners  are  unable  to  get  the  fact-feature  cross- 
reference. 

It  is  interesting  to  note  that  the  interference  effect  is  seen  only  in  the  inference  test  and  name  recall  and  not 
in  fact  recall  and  name  to  fact  match.  Though  the  trend  is  maintained,  it  does  not  reach  the  level  of  significance. 

Educational  Importance 

The  results  of  this  study  have  some  practical  implications  in  the  use  of  audio  for  educators  interested  in 
designing  online  instruction  involving  geographic  maps.  Online  instruction  containing  geographic  maps  should 
employ  the  computer’s  capability  to  offer  different  modalities  (audio  and  dual)  to  enhance  learning.  This  study  also 
provides  evidence  that  using  both  audio  and  text  to  present  verbal  information  may  be  more  effective  than  using 
audio  alone  in  single  temporal  contiguity  conditions  while  the  reverse  is  true  for  dual  contiguity  (temporal  and 
spatial  contiguities)  with  single  modality. 

With  digital  books  becoming  more  and  more  common  this  research  has  lots  of  implications  as  it  could  lead 
to  better  use  of  technology  for  improving  standards  of  education. 
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Singin'  the  Blues  On  the  ICN 
A Study  In  Applied  Music  Distance  Education 

David  Stuart 
Donald  Simonson 
Iowa  State  University 


As  the  21st  Century  progresses,  music  teachers  and/or  coaches  are  no  longer  bound  to  the  confines  of  their 
studios.  With  the  advent  of  distance  education  technology  and  the  Internet,  lessons  and  coachings  can  be  conducted 
over  hundreds  or  thousands  of  miles,  or  even  across  a continent. 

Since  1993,  at  Iowa  State  University,  hundreds  of  students  have  participated  in  master  classes,  lessons  and 
coachings  at  a distance.  The  Iowa  Communication  Network  (ICN)  connects  over  750  sites  in  the  state  to  comparable 
sites  across  America  and  throughout  the  world.  Users  employ  a fully  interactive,  broadcast  quality  audio -video, 
digital  fiber-optic  network,  capable  of  connecting  hundreds  of  sites  simultaneously.  Sites  are  conveniently  located  in 
public  schools,  colleges,  universities,  and  additional  governmental,  educational  and  business  locations.  The  cost  is 
reasonable  and  is  often  subsidized  by  the  state.  In  Iowa,  participants  pay  only  a small  $5.00-$10.00  per  hour 
connection  fee  per  site.  For  multiple  participant  activities  such  as  master  classes,  the  technology  is  particularly  cost 
effective. 

By  employing  the  ICN,  students  can  gain  access  to  master  teachers  and  coaches  hundreds  of  miles  away 
and  interact  with  them  as  if  they  were  in  the  same  studio. 

To  examine  the  effectiveness  of  distance  education  technology  in  applied  music,  participants  in  ICN 
lessons  and  master  classes  were  given  brief  pre-  and  post  experience  surveys.  The  results  are  presented  below: 


♦ Do  you  believe  it  is  feasible  to  teach  (or  learn)  applied  music  instruction  at  a distance  (asked  prior  to 
the  distance  learning  experience)? 

No  44 

Yes  25 


♦ What  do  you  believe  will  be  the  greatest  drawbacks  to  distance  learning  in  applied  music  instruction 


(pre -experience)? 

Sound  Quality  31 

Video  Quality  1 

Ease  of  Use  9 

Classroom/Student  Management  7 

Interactivity  19 

Other  7 


♦ What  were  the  greatest  drawbacks  to  distance  learning  in  applied  music  instruction  (post-experience)? 


Sound  Quality  15 

Video  Quality  1 

Ease  of  Use  5 

Classroom/Student  Management  7 

Interactivity  13 

Other  27 


♦ What  do  you  believe  will  be  the  greatest  benefits  in  distance  learning  in  applied  music  instruction  (pre  - 


experience)? 

Access  to  Experts  51 

Interactivity  10 

Ease  of  Use  5 

Low  Cost  1 

Increased  Feedback  12 

Other  3 
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♦ What  were  the  greatest  benefits  of  distance  learning  in  applied  music  instruction  (post-experience)? 


Access  to  Experts  51 

Interactivity  13 

Ease  of  Use  10 

Low  Cost  2 

Increased  Feedback  13 

Other  3 


♦ Please  rate  the  quality  of  the  sound  on  a scale  of  1 - 5 with  5 as  the  highest. 
Speech  Sound  Quality  4.2 

Music  Sound  Quality  4.03 

Overall  Sound  Quality  4. 1 1 


♦ Please  rate  the  quality  of  the  video  image  on  a scale  of  1 - 5 with  5 as  the  highest. 
Video  Quality  of  Print  Media 
And  Graphics  4.31 

Video  Quality  of  Teacher 
And  Performers  4.45 

Overall  Video  Quality  4.38 


♦ Please  rate  the  ease  of  use  of  the  distance  learning  technology  on  a scale  of  1 - 5 with  5 as  highest. 
Ease  of  Use  as  an  Active 
Participant  4.38 

Ease  of  Use  as  an  Auditor  4.38 

Overall  Ease  of  Use  4.38 


♦ How  effective  did  you  believe  applied  voice  distance  teaching  (learning)  would  be  on  a scale  of  1 - 5 
with  5 as  the  highest? 

Expected  Effectiveness  3.66 

♦ How  effective  was  applied  voice  distance  teaching  (learning)  would  be  on  a scale  of 
1-5  with  5 as  the  highest? 

Actual  Effectiveness  4.44 


♦ Compared  to  other  forms  of  learning  how  do  you  rank  your  applied  music  instruction  distance 
learning  experience  on  a scale  of  1 - 5 with  5 as  the  highest? 

Comparison  to  Other  Forms 

Of  Learning  4.43 

Is  teaching/coaching/learning  singing  at  a distance  a viable  methodology?  Does  it  really  work? 
Unequivocally,  YES  and  YES ! 

When  should  this  technology  be  used? 

Whenever  circumstances  inhibit  or  prevent  face-to-face  learning! 

For  example: 

• Master  Class  Broadcasting 

• Continuing  Education  or  In-Service  Workshops 

• Special  Topics  Workshops  or  Coaching 

• Outreach  And  Extension  Activities 

• Recruiting  Activities 

• When  Travel  Is  Prohibited 

• Other? 
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What  does  the  future  hold  for  distance  learning  technology? 

• Within  years  a widely  available,  cost  effective  PC  based  Internet  Delivery  Systems. 

“Applied  Music  Distance  Education”  will  be  commonplace,  with  the  technology  as  close  as  your  laptop  computer. 
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The  Effect  Of  Oral  Description  In  Complementing  Animated  Instruction  In  A 
Web-based  Learning  Environment  On  Undergraduate  Students  Achievement 

Of  Different  Educational  Objectives 


Jeff  Swain 
Mine  Munyofu 
Khusro  Kidwai 
Huifen  Lin 
Bradley  Ausman 
Francis  Dwyer 

The  Pennsylvania  State  University 

Abstract 

The  purpose  of  this  study  was  to  test  the  principle  of  modality  by  using  audio  to  deliver  verbal  information 
when  that  information  is  designed  to  support  non-verbal  information  such  as  animations  in  a computer-based 
lesson.  This  was  done  by  comparing  the  effect  of  two  types  of  audio  support  mechanisms-  a simple  support 
mechanism  consisting  of  declarative  statements  explaining  the  animated  sequence  and  a complex  support 
mechanism  consisting  of  questions  and  answers  explaining  the  animated  sequence-  on  undergraduate  student 
achievement  of  conceptual,  rule  and  procedure  knowledge.  A control  group  consisting  of  the  same  computer-based 
lesson  without  any  auditory  support  of  the  animation  was  also  employed.  Learning  was  measured  through  drawing, 
terminology,  and  comprehension  tests.  The  results  indicate  student  achievement  was  not  enhanced  by  the  addition  of 
auditory  support. 


Introduction 

With  computers  coming  equipped  with  multi- media  features  such  as  the  ability  to  play  sound  and 
animation,  teachers  and  instructional  designers  are  able  to  develop  and  deliver  lessons  in  novel  ways  using  both  the 
visual  and  auditory  channels  of  students.  But,  is  student  performance  improved  by  the  use  of  both  channels  as 
opposed  to  the  traditional  delivery  method  relying  solely  on  the  visual  channel?  And,  at  what  learning  level  (the 
factual,  conceptual,  rule  & procedure  level)  is  there  improvement  in  performance  when  both  channels  are  used? 

According  to  cognitive  load  theory  (Kalyuga,  Chandler  and  Sweller  1988,  1989;  Sweller  1988;  Baddeley 
1986,  1992)  methods  of  instruction  reducing  working  memory  load  in  order  facilitate  the  encoding  and  storing  of  the 
information  in  long-term  memory  are  effective.  One  such  method  is  dual  coding  theory  (Sadoski  and  Paivio  2001; 
Clark  and  Paivio  1991;  Paivio  1971,  1986,  1990).  Dual  coding  theory  assumes  we  have  two  information  processing 
systems:  a verbal  system,  comprised  of  words,  whose  strength  lies  in  its  sequentially  ordered  hierarchy,  each  bit  of 
information  paves  the  way  for  the  next,  and  a non-verbal  system  whose  strength  lies  in  its  synchronous  (holistic) 
hierarchy. 

Using  audio  to  deliver  verbal  information  when  that  information  is  designed  to  support  non-verbal 
information  such  as  graphics,  pictures,  and  animations  can  enhance  the  effect  of  using  both  the  verbal  and  visual 
systems.  This  is  known  as  the  modality  effect  (Clark  & Mayer  2003;  Penney  1989;  Paivio  1986).  The  modality 
effect  (Clark  & Mayer,  2003,  pp.  93)  states  “people  learn  more  deeply  from  multimedia  lessons  when  words 
explaining  concurrent  animations  or  graphics  are  presented  as  speech  rather  than  as  onscreen  text.” 

The  studies  reviewed  indicate  that  the  modality  effect  works  well  at  improving  student’s  verbal  recall  of 
factual  information  (Mayer  1991,  Mayer  1992,  Barron  1993,  Mousavi  1995,  Mann  1995,  Mayer  1996,  Mayer  1998, 
Moreno  1999,  Mayer  2001,  and  Moreno  2002).  There  is  also  indication  that  the  modality  effect  works  at  improving 
student’s  ability  to  solve  problems  (Mayer  1991,  Mayer  1992,  Barron  1993,  Mayer  1994,  Mousavi  1995,  Mayer 
1996,  Mayer  1998,  Moreno  1999,  Chuang  1999,  Mayer  2001,  Moreno  2002).  However,  there  is  limited  information 
regarding  the  effect  of  modality  of  student  achievement  of  learning  concepts,  rules  and  procedures.  The  studies 
conducted  by  Mayer  (1998)  and  Moreno  (1999,  2002)  found  evidence  to  support  the  positive  effect  of  modality  on 
student’s  ability  to  learn  conceptual  information  but  these  learning  levels  were  not  isolated  and  studied  on  their  own. 
In  these  studies,  student’s  ability  to  recall  facts,  identify  concepts,  and  solve  problems  were  tested  together.  This 
leaves  open  the  question  of  how  effective  is  the  use  of  modality  if  the  goal  of  the  lesson  is  to  facilitate  achievement 
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of  conceptual  and  rule/procedure  knowledge?  This  study  seeks  to  begin  filling  in  the  gap  in  the  literature  by 
isolating  these  two  intellectual  skill  learning-levels. 

Literature  Review 

Studies  Exploring  The  Effect  Of  Dual-Coding:  Using  The  Visual  And  Verbal  Channels 

Mayer,  in  his  study  on  how  computer  based  animations  can  be  used  to  promote  scientific  understanding 
(1991)  and  in  his  study  aimed  at  identifying  the  role  of  student’s  spatial  ability  in  learning  from  words  and  pictures 
(1994),  found  that  undergraduate  college  students,  given  a lesson  in  an  area  of  non-expertise,  performed  better  on 
recall  and  problem  solving  tests  when  both  the  verbal  and  visual  systems  were  utilized.  Mayer  also  found  that  the 
effect  of  dual  coding  was  enhanced  when  the  verbal  and  visual  information  was  presented  concurrently,  at  the  same 
time  as  the  animation  rather  than  before  or  after  it  (Mayer  1991,  1992,  1994).  This  finding  was  duplicated  by 
Moreno  (1999)  who  tested  undergraduate  college  students  with  low  prior  knowledge  of  meteorology,  ability  to 
recall  information  about  the  process  of  lightning.  Chuang  (1999)  found  similar  results  working  with  seventh  grade 
students  in  Taipei,  Taiwan  when  studying  the  role  of  gender  and  field  dependence/independence  on  the  ability  to 
solve  math  problems. 

Placing  supporting  text  near  the  animation  it  is  meant  to  support  in  known  as  the  contiguity  principle  (Clark 
and  Mayer  2000)  or  the  split-attention  effect  (Chandler  and  Sweller  19992;  Sweller,  Chandler,  Tierney,  and  Cooper 
1990;  Tarmizi  and  Sweller  1988).  The  split -attention  affect  happens  when  students  must  divide  their  attention 
between  multiple  sources  of  information  (Mousavi  1995).  The  split -attention  effect  can  be  reduced  by  placing 
printed  words  next  to  the  animation  they  are  supporting  (Clark  and  Mayer  2000). 

The  positive  effect  of  dual  coding  in  reducing  cognitive  load  also  was  evident  in  studies  exploring  the 
impact  of  reducing  cognitive  load  in  the  lesson  summary.  In  his  study  to  see  if  reducing  cognitive  load  in  lesson 
summaries  would  help  increase  student’s  retention,  Mayer  (1996)  found  that  undergraduate  students  performed 
better  on  tests  of  recall  and  problem  solving  when  summaries  included  both  illustrations  and  text. 

The  results  indicate  the  effectiveness  of  verbal,  in  the  form  of  text,  support  of  animation  in  reducing 
cognitive  load.  Animation  complemented  with  a textual  explanation  enabled  students  to  take  greater  advantage  of 
their  capability  to  process  information  on  two  levels  by  stimulating  the  visual  system  and  by  reducing  the  load 
placed  on  the  verbal  processing  system.  This  re -shuffling  of  information  in  working  memory  increased  their  ability 
to  make  meaning  out  of  the  information  in  preparation  for  storage  in  long-term  memory.  The  placement  of  the 
supporting  textual  explanation  next  to  the  animation  further  reduced  cognitive  load  and  enhanced  performance. 

Studies  Exploring  The  Effect  Of  Modality:  Using  The  Spoken  Word  In  Place  Of  The  Written  Word  To 
Support  The  Visual  Channel 

While  animation  helped  to  reduce  cognitive  load  it  was  not  reduced  as  much  as  it  could  be  because  both 
text  and  animation  have  to  pass  through  the  same  (the  visual)  sensory  channel  (Mousavi  1995;  Chandler  and  Sweller 
1992.  This  meant  that  students  were  forced  to  shift  their  attention  between  the  text  and  the  animation  while  going 
through  the  pattern  recognition  and  selective  perception  processes.  Miller  (1956,  pp.  85)  referred  to  the  limitation  of 
the  sensory  register  as  our  “channel  capacity.’’  Channel  capacity  is  the  maximum  amount  of  information  we  can  hold 
in  our  sensory  memory  at  any  given  point  in  time. 

When  animation  is  supported  by  a spoken  explanation,  as  opposed  to  a textual  explanation,  cognitive  load 
is  further  reduced.  This  time  the  reduction  comes  through  the  way  that  information  passes  from  the  environment 
through  sensory  memory  and  into  working  memory  (Chandler  and  Sweller  1992;  Paivio  1986;  Penney  1989). 

Mann  (1995),  in  as  study  testing  student’s  ability  to  construct  a solution  to  an  educational  problem,  used 
temporal  sound,  spoken  information  that  highlights  or  details  static  or  moving  visuals,  in  a computer-based  lesson  to 
more  effectively  use  student’s  channel  capacity  in  sensory  memory  by  using  sound  with  text  to  support  concurrent 
animation.  Mann  found  that  students  were  able  to  recall  a greater  amount  of  critical  detail  when  temporal  sound  was 
used. 

While  Mann  used  both  written  text  and  temporal  sound  simultaneously,  Mayer  (1998),  in  a study  that  tested 
student’s  ability  to  recall  how  lightening  works,  compared  the  effect  of  using  either  temporal  sound  or  written  text  to 
support  concurrent  animation  and  found  that  students  were  able  to  recall  more  information  and  perform  better  on 
problem  solving  tests  if  they  received  the  lesson  using  temporal  sound. 

Lai  (2000),  in  her  study  testing  the  evarious  types  of  visual  illustrations  on  concept  learning,  also  found  that 
students  receiving  lessons  using  temporal  sound  to  support  static  graphics  performed  better  on  matching  tests  than 
students  receiving  lessons  that  used  text  to  perform  the  same  function. 

Similar  to  spatial  contiguity,  the  placement  of  text  near  the  animation  it  supports,  the  contiguity  affect  also 
applies  when  temporal  sound  is  used  to  support  animation.  Temporal  contiguity  occurs  when  visual  and  spoken 
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materials  are  presented  simultaneously  rather  than  successively  (Moreno  1999).  Moreno  (1999),  using  a lesson  on 
the  process  of  lightning  formation,  found  that  learning  was  negatively  impacted  if  the  temporal  sound  was  not 
placed  concurrent  with  the  animation  it  supported  matching  Mayer’s  (1994)  findings  with  written  text. 

Using  written  text  or  temporal  sound  that  closely  matches  the  animation  it  supports  without  a lot  of 
extraneous  details  is  also  an  important  factor  for  success.  In  a study  on  reducing  the  cognitive  load  in  lesson 
summaries,  Mayer  (1996),  using  a lesson  on  the  process  of  lightning  formation,  found  that  students  receiving 
concise  lesson  summaries,  that  included  both  visual  and  verbal  information,  performed  best  on  verbal  recall  and 
problem  solving  tests.  Similarly,  Moreno  (2002)  found  that  extraneous  details  hurt  student  performance  when  using 
the  lesson  on  the  process  of  lightning  formation. 

Issues  Of  Instructional  Consistency 

Instructional  consistency  (Canelos  1983;  Gagne  and  Medsker  1996))  states  that  intellectual  skills  are 
hierarchical  in  nature  and  that  lower  order  skills  are  prerequisite  for  learning  higher  order  skills.  Verbal  skills 
comprise  the  base  of  the  hierarchy  and  are  the  ability  to  recall  factual  information.  Verbal  skills  are  a prerequisite  for 
learning  discriminations.  Discriminations,  the  ability  to  distinguish  between  things,  come  next  and  are  a prerequisite 
for  concept  formation.  Concepts  are  the  ability  to  classify  information  based  on  its  critical  attributes  and  are  a 
prerequisite  for  the  learning  of  rules,  which  are  the  ability  to  specify  the  relationship  between  concepts.  At  the  top  of 
the  hierarchy  sits  higher  order  rules,  the  ability  to  use  multiple  rules  in  order  to  perform  a task  or  solve  a problem 
(Gagne  and  Medsker  1996,  pp.  32-33). 

Therefore,  if  the  objective  is  to  generate  solutions  the  instructional  unit  must  contain  the  rules/procedures, 
concepts,  and  facts  that  represent  the  prerequisite  knowledge  needed  to  solve  the  problem.  In  the  studies  reviewed 
animation  supported  by  narration  increased  students  ability  to  recall  factual  information  and  to  generate  solutions  to 
problems.  The  majority  of  studies  reviewed  explored  student’s  ability  to  recall  factual  information  and  solve 
problems  (Mayer  1991,  1992  1996;  Mousavi  1995),  to  solve  problems  (Barron  1993;  Mayer  19994,  2001),  or  to 
recall  information  (Mann  1995).  These  studies  do  not  give  us  an  indication  of  where  or  how  animation  supported  by 
narration  works  in  the  instructional  hierarchy.  Is  animation  supported  by  concurrent  narration  effective  at  teaching 
concepts  or  rules/principles?  Or  is  there  something  about  the  animation  with  narration  that  enables  students  to  build 
connections  among  intellectual  skills? 

This  study  aims  to  build  on  the  existing  knowledge  base  and  begin  to  fill  in  the  gaps  in  the  literature  by 
testing  the  hypothesis  that  animation  supported  by  concurrent  temporal  sound  is  better  at  teaching  concepts  and 
rules/principles  than  animation  alone. 


Purpose  Of  This  Study 

The  purpose  of  this  study  was  to  test  the  principle  of  modality  by  using  audio  to  deliver  verbal  information 
when  that  information  is  designed  to  support  non-verbal  information  such  as  animations  in  a computer-based  lesson. 
This  was  done  by  comparing  the  effect  of  two  types  of  audio  support  mechanisms  - a simple  support  mechanism 
consisting  of  declarative  statements  explaining  the  animated  sequence  and  a complex  support  mechanism  consisting 
of  questions  and  answers  explaining  the  animated  sequence  - on  undergraduate  student  achievement  of  conceptual, 
rule  and  procedure  knowledge.  The  questions  the  current  study  seeks  to  provide  insight  into  include; 

1.  Do  students  receiving  the  treatment  consisting  of  the  simple  audio  support  (declarative  statements 
explaining  the  animation)  perform  better  on  tests  of  conceptual  and  rule  & procedural  knowledge 
than  students  receiving  the  treatment  with  animation  alone? 

2.  Do  students  receiving  the  treatment  consisting  of  complex  audio  support  (questions  followed  up 
with  declarative  statements  explaining  the  animation)  perform  better  on  tests  of  conceptual  and 
rule  & procedural  knowledge  than  students  receiving  the  simple  audio  treatment  and  the  treatment 
consisting  of  animation  alone? 

Research  Design  And  Methodology 

A posttest  only  design  method  was  used.  The  placement  and  use  of  animation  and  temporal  sound  was 
derived  based  on  the  results  of  two  pilot  studies.  In  the  first  study  a computer-based  lesson  using  Dwyer’s  (1977) 
lesson  on  the  human  heart,  titled  “The  Heart  And  Its  Functions”,  was  used  to  determine  where  to  place  the 
animation.  Since  the  goal  of  the  primary  study  was  to  test  student’s  ability  to  perform  on  tests  of  conceptual  and 
rule/procedure  knowledge,  the  lesson  was  designed  using  programmed  instruction  to  ensure  that  adequate  factual 
knowledge  was  achieved.  An  item  analysis  was  completed  to  determine  where  to  place  the  animation  in  the  lesson 
for  the  subsequent  study.  A difficulty  level  of  .60  was  used  as  the  cutoff  meaning  that  any  item  with  a difficulty 
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level  below  sixty  percent  was  targeted  for  animated  support. 

A series  of  four  tests  consisting  of  20  questions  each  were  used  to  assess  student  achievement.  An 
identification  test  was  used  to  assess  factual  knowledge.  A drawing  test  and  a terminology  test  were  used  to  measure 
conceptual  knowledge.  A comprehension  test  was  used  to  measure  rule/procedure  knowledge.  The  validity  and 
reliability  of  the  tests  was  reported  by  Dwyer  and  Moore  (1978).  Based  on  the  outcomes  of  the  pilot  study,  18 
animations  were  developed  and  placed  in  the  lesson  adjacent  to  the  textual  material  they  were  designed  to  support. 

A second  pilot  study  was  conducted  in  order  to  determine  which  of  the  animated  sequences  required  the  use 
of  temporal  support.  Again,  item  difficulty  was  set  at  .60  with  any  items  supported  by  animation  scoring  below  sixty 
percent  targeted  for  support  using  temporal  sound.  The  same  four  tests  were  used  to  assess  student  achievement. 
Based  on  the  results  of  this  analysis  two  treatments,  one  using  simple  audio  support  and  another  using  complex 
audio  support,  were  developed  to  support  the  animations. 

For  the  primary  study,  eighty-eight  undergraduate  students  were  recruited  from  a management  class,  an 
educational  psychology  class,  and  an  information  systems  class.  These  students  were  randomly  assigned  to  one  of 
three  experimental  groups:  A control  group  that  received  the  lesson  with  animation  but  no  audio  (NA)  support.  A 
treatment  group  assigned  a lesson  that  used  simple  audio  (SA)  explanations,  in  the  form  of  declarative  sentences,  in 
support  of  the  animation.  And,  a treatment  group  assigned  a lesson  that  used  complex  audio  (CA)  explanations,  in 
the  form  of  Questions  and  answers,  in  support  of  the  animation.  Twenty-nine  students  received  the  treatment  with  no 
audio  support.  Thirty  students  received  the  treatment  with  simple  audio  support.  Twenty-nine  students  received  the 
treatment  with  the  complex  audio  support.  All  students  received  extra  credit  towards  their  final  grade  in  the  class  for 
participating  in  the  study. 

All  three  experimental  groups  (NA,  SA,  CA)  received  a treatment  where  the  beginning  of  the  lesson  was 
comprised  of  programmed  instruction  to  ensure  the  prerequisite  factual  knowledge  was  gained.  However,  due  to  the 
nature  of  the  content  it  was  impossible  to  deliver  the  lesson  with  each  learning  level  isolated.  Therefore,  the 
programmed  instruction  section  also  contained  conceptual  information  along  with  the  factual  information.  The 
programmed  instruction  consisted  of  a web  page  containing  one  or  two  pieces  of  factual  knowledge.  This  meant  that 
these  pages  also  contained  animation  or  animation  with  temporal  support  if  the  item  analysis  indicated  it  was 
needed. 

After  a series  of  three  to  four  pages  like  this,  students  were  asked  to  answer  a series  of  practice  questions 
based  on  the  material  just  presented.  If  the  student’s  score  was  satisfactory  they  were  able  to  move  on  to  the  next 
part  of  the  lesson.  If  the  score  was  unsatisfactory  they  student  was  brought  back  to  the  beginning  of  that  series  of 
content.  There  was  no  limit  placed  on  the  amount  of  time  or  the  number  of  times  the  student  could  spend  on  one 
section.  Once  the  student  satisfactorily  completed  the  programmed  instruction  segment  they  were  given  a pencil  and 
paper  drawing  test  in  which  they  were  asked  to  draw  and  label  the  main  sections  of  the  human  heart.  Once  that  was 
completed  the  students  were  asked  to  complete  an  online  identification  test.  In  this  test  a picture  of  a heart  was 
presented  with  an  arrow  pointing  to  the  section  to  be  identified.  Students  were  asked  to  select  the  name  of  the 
section  from  a list  of  four  choices. 

The  second  part  of  the  lesson  was  primarily  focused  on  rule/procedure  information  although  there  was  also 
some  conceptual  information  presented.  Students  went  through  a series  of  web  pages  describing  the  flow  of  blood 
as  it  passes  through  the  heart.  Some  pages  contained  animated  support  of  the  content  and  some  pages  contained 
animation  along  with  temporal  support.  The  control  group  (NA)  lesson  contained  only  animation.  Where  the  SA 
group  received  temporal  support  for  animation  it  was  in  the  form  of  simple  declarative  sentences.  For  example,  if 
the  animation  were  designed  to  show  that  the  ventricles  are  the  thickest  walled  chambers  of  the  heart,  the  student, 
when  he/she  selected  the  play  button  would  see  the  animation  and  simultaneously  hear  a statement  that  said,  “The 
ventricles  are  the  thickest  walled  chambers  of  the  heart.” 

The  CA  group  received  the  same  treatment  as  the  SA  group  except  that  the  temporal  support  was  delivered 
in  a question  and  answer  format.  Continuing  the  example  above  regarding  the  ventricles,  a student  in  the  CA  group 
would  hear  while  the  animation  was  playing,  “What  are  the  thickest  walled  chambers  of  the  heart?  The  ventricles  are 
the  thickest  walled  chambers  of  the  heart.” 

The  same  voice  was  used  in  both  treatment  groups  and  the  speed  at  which  the  animation  was  played  was 
adjusted  to  fit  the  length  of  the  temporal  support.  In  all  three  experimental  groups,  students  were  allowed  to  replay 
the  animation,  as  many  times  as  they  felt  was  necessary  to  understand  the  material. 

At  the  end  of  the  lesson  students  in  each  group  were  asked  to  complete  a terminology  test  where  they  were 
asked  to  complete  a sentence  by  selecting  the  appropriate  word  or  phrase  from  the  choices  provided,  and  a 
comprehension  test  where  they  were  asked  to  answer  a question  by  selecting  the  appropriate  answer  from  a list  of 
choices.  The  identification,  terminology,  and  comprehension  tests  were  built  into  the  computer-based  lesson  and 
completed  on-line. 
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Results  And  Implications 

ANOVA  was  conducted  to  compare  the  differences  between  the  three  experimental  groups  on  scores  on  the 
four  criterion  tests.  Alpha  was  set  at  the  .05  level.  Comparisons  were  made  at  two  levels:  using  all  items  and  using 
only  the  items  identified  by  the  item  analysis.  Comparisons  were  made  using  all  four  tests. 


Results  Using  All  Items 


The  table  below  details  the  mean  and  standard  deviation  for  each  treatment  group  counting  all  items 


Mean 

Std. 

Deviation 

drawing  test 

control 

15.97 

3.257 

simple 

16.00 

3.28C 

complex 

17.31 

2.714 

Total 

16.42 

3.125 

identification  test 

control 

17.62 

2.47C 

simple 

17.40 

2.094 

complex 

17.88 

2.215 

Total 

17.62 

2.247 

terminology  test 

control 

11.93 

5.028 

simple 

12.70 

4.17C 

complex 

12.58 

5.077 

Total 

12.40 

4.714 

comprehension 

control 

11.00 

3.464 

test 

simple 

11.27 

3.60C 

complex 

12.08 

4.06S 

Total 

11.42 

3.688 

Test  Total 

control 

56.52 

11.903 

simple 

57.37 

10.545 

complex 

59.73 

9.298 

Total 

57.80 

10.637 

The  F statistic  for  the  drawing  test  was  1.787;  the  F statistic  for  the  identification  test  was  .319;  the  F 
statistic  for  the  terminology  test  was  .218;  the  F statistic  for  the  comprehension  test  was  .621.  The  F statistic  for  the 
total  of  all  tests  was  .659.  These  results  indicate  that  there  were  no  significant  differences  in  the  performance  of 
students  in  each  group.  The  reliability  coefficient  was  .8612. 


Results  Using  Only  Items  Identified  Through  Item  Analysis 

The  table  below  details  the  mean  and  standard  deviation  for  each  treatment  group  for  only  the  items 
identified  as  deficient  through  item  analysis 


Mean 

Std. 

Deviation 

Drawing  Test 

control 

3.28 

1.601 

simple 

2.97 

1.450 

complex 

3.93 

1.163 

Total 

3.39 

1.458 

Terminology 

control 

4.14 

2.489 

Test 

simple 

4.93 

2.243 

complex 

4.73 

2.539 

Total 

4.60 

2.416 

Comprehension 

control 

4.38 

1.635 
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Test 


Item  Total 


simple 

4.60 

2.010 

complex 

4.69 

2.074 

Total 

4.55 

1.893 

control 

11.79 

4.378 

simple 

12.5C 

4.392 

complex 

13.27 

3.853 

Total 

12.49 

4.222 

The  F statistic  for  the  drawing  test  was  3.547;  the  F statistic  for  the  identification  test  was  .851;  the  F 
statistic  for  the  comprehension  test  was  .198.  The  F statistic  for  the  item  total  on  these  tests  was  .835.  These  results 
indicate  that  there  were  no  significant  differences  in  the  performance  of  students  in  each  group.  The  reliability 
coefficient  was  .7778. 

Implications  Of  The  Results 

The  lack  of  significance  in  performance  between  the  experimental  groups  indicates  that  using  either 
animation  or  animation  along  with  temporal  support  may  be  problematic  when  it  comes  to  teaching  concepts  and 
rules/principles.  These  results  are  interesting  in  lieu  of  prior  studies  whose  results  indicated  animation  supported 
with  temporal  sound  was  effective  at  teaching  facts  and  problem-solving  skills.  With  factual  knowledge  being  a 
prerequisite  for  learning  concepts,  rules/principles  and  rules/principle  knowledge  being  a prerequisite  for  problem 
solving  learning  it  was  thought  that  animation  supported  by  temporal  sound  would  have  been  effective  at  teaching 
concepts  and  rules/principles.  However,  the  results  of  this  study  do  not  support  this  hypothesis. 

These  results  do,  however,  suggest  possibilities  for  further  research.  Some  possible  avenues  for  exploration 
include:  What  is  it,  of  there  is  anything,  about  the  nature  of  concepts  and  rules/procedures  that  may  not  make  them 
amenable  to  animation  based  learning?  Does  temporal  sound  increase  cognitive  load  when  the  lesson  is  aimed  at 
conceptual  or  rule/principle  information?  Does  the  kind  of  content  being  presented  limit  the  effectiveness  of 
modality?  Are  methods  other  than  simple  declarative  sentences  or  questions  followed  by  answers  better  suited  for 
teaching  concepts  and  rules/procedures? 


Summary 

Using  temporal  sound  to  support  animation  in  computer-based  lessons  has  been  effective  when  the  goal  is 
to  teach  factual  knowledge.  There  is  also  indication  that  it  is  effective  for  teaching  problem-solving  skills.  This 
study,  however,  did  not  find  evidence  to  support  that  using  animation  with  temporal  sound  to  teach  concepts, 
rules/principles  is  effective.  The  results  indicate  that  there  is  a problem  of  instructional  consistency  when  applying 
the  modality  effect  to  the  learning  of  intellectual  skills. 
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Using  Course  Syllabi  to  Determine  Amount  of  Student  Exposure  to  and  Use  of 
Technology  in  Instruction:  PT3  Assessment 

Berhane  Teclehaimanot 
University  of  Toledo 

Abstract 

During  the  spring  semester  2003,  faculty  members  in  the  Colleges  of  Education  and  Arts  & Sciences  were 
asked  to  submit  their  fall  2002  semester  syllabi  that  reflect  the  integration  of  technology  into  the  courses  they  teach 
as  a result  of  attending  12-15  weeks  faculty  development/training  workshops.  Furthermore,  a survey  was  designed 
to  extract  information  as  to  what  extent  faculty  use  technology  into  their  courses  and  to  what  extent  they  require 
their  students  to  use  technology  in  the  classroom.  During  the  2001  -02  academic  year,  32  faculty  members  from  the 
Colleges  Education  and  Arts  and  Sciences  participated  the  PT 3 faculty  development  implementation  grant 
workshops.  The  purpose  of  this  study  is  to  share  our  experiences  related  to  this  faculty  development  effort  during  the 
first  year. 


Background 

During  the  spring  semester  of  2002,  University  of  Toledo  began  its  first  group  faculty  development/training 
workshops  to  prepare  faculty  to  integrate  technology  into  the  college  classrooms.  The  main  goals  and  objectives  of 
the  project  were  to  enhance  student  learning  simply  teaching  the  use  of  technology  tools  to  pre -service  teachers,  and 
also  by  modeling  the  application  and  integration  processes.  Preservice  teachers  must  be  shown,  systematic  approach, 
how  to  integrate  technology  and  eventually  exhibit  these  same  processes  them  selves.  It  is  this  teaching  and 
modeling  of  best  practices  that  cements  a pre  -service  teacher’ s commitment  to  the  application  and  adaptation  of 
technology  with  the  expectation  that  it  will  carry  over  to  the  classroom  of  the  new  teacher. 

The  integration  of  technology  into  college-level  teacher  education  curriculum  has  become  a catchphrase  in 
both  higher  and  K-12  education.  Unfortunately,  the  missing  element  in  this  technological  innovation  seems  to  be 
faculty  development.  While  most  schools  recognize  the  need  for  students  to  master  and  utilize  technology,  the 
missing  element  in  this  technological  innovation  is  in  the  area  of  faculty  development.  Although  most  faculty 
recognize  the  integration  of  technology  into  the  curriculum  requires  new  approaches  to  the  teaching  and  learning 
process,  not  enough  consideration  is  being  paid  to  ensuring  that  the  faculty  who  are  teaching  with  the  new 
innovative  technologies  have  been  adequately  trained  (Rowe,  1999).  In  order  to  facilitate  faculty  in  effectively 
integrating  technology  across  curriculum  in  the  classroom,  the  U.S.  Department  of  Education  “Preparing 
Tomorrow’s  Teachers  to  Use  Technology  (PT3)  implementation  grant  projects  are  playing  an  essential  role  in 
disseminating  information  to  ensure  that  our  future  K-12  teachers  are  technology-wise  and  can  use  the  multimedia 
facilities  available  in  today's  schools.  Ultimately,  upon  graduation,  these  future  teachers  will  not  only  practice  their 
skills  in  the  classroom  but  will  model  the  effective  use  of  technology  to  their  students.  First,  faculty  development 
and  training  must  be  available  and  must  include  the  most  current  technological  tools  in  order  to  provide  an 
understanding  of  how  to  integrate  the  effective  use  of  technology  into  the  teaching  and  student  learning  process. 

In  spite  of  the  tremendous  growth  of  technology  in  our  nation’s  schools  and  the  belief  by  a majority  of 
educators  that  all  students  must  have  access  to  technology  to  be  truly  successful  in  today’s  world,  there  is  evidence 
that  many  teachers  still  do  not  use  technology  at  all  in  their  teaching  (Education  Week,  1999).  According  to  a 
national  survey  (DeMedio  & Teclehaimanot,  2001),  90  percent  of  schools  in  ten  states  have  Internet  access  to 
classroom.  In  addition,  according  to  a report  from  the  research  of  the  CEO  Forum  on  Education  and  Technology 
(2000),  80%  of  schools  in  the  United  States  have  access  to  technology,  but  few  teachers  are  ready  to  use  technology 
in  their  classroom  activities.  The  Department  of  Education  Survey  found  that  approximately  33%  of  teachers 
believe  that  they  could  use  technology  in  their  classrooms  (National  Center  for  Education  Statistics,  2000).  A 
national  survey  by  Backer  (1999)  supports  the  finding  that  majorities  of  teachers,  as  many  as  70%,  are  not  using  the 
technologies  available  to  them. 

Faculty  belief  and  attitude  toward  technology,  fear  factors  or  complexity  issues,  lack  of  time  and  a lack  of 
support,  limited  access,  lack  of  faculty  development  training  and  lack  of  organizational  support  have  been  identified 
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as  major  barriers.  There  are  many  factors  to  change,  such  as  lack  of  computers  in  the  classroom,  access  to  the 
Internet,  updated  software  and  hardware,  lack  of  release  time  for  faculty  to  learn  how  to  integrate  technology  into 
the  curriculum.  Teachers  must  be  allowed  adequate  time  to  learn  new  technologies  (Maney,  1999).  While  numerous 
studies  have  examined  the  use  of  technology  in  the  K-12  classrooms,  few  have  taken  the  focus  one  step  further  to 
determine  the  extent,  to  which  technology  is  used  and  required  in  the  undergraduate  teacher  education  programs. 

The  purpose  of  this  study  is  to  share  our  experiences  related  to  this  faculty  development  effort  during  the  first  year. 

Faculty  members  of  the  spring  2002  TIPT  technology  workshops  were  asked  to  submit  fall  2002  semester 
syllabi  that  reflected  the  integration  of  technology  into  their  courses  as  a result  of  attending  these  workshops.  Of  the 
30  faculty  participants,  20  returned  syllabi  to  date  and  5 indicated  that  they  were  either  not  currently  teaching  the 
courses  they  had  adapted  or  they  were  away  on  sabbatical.  Syllabi  from  these  5 will  be  collected  in  spring  2003 
along  with  those  of  the  fall  2002  participants.  Syllabi  from  the  remaining  5 participants  are  still  being  solicited. 

Design 

The  rubric  used  to  measure  technology  use/exposure  can  be  found  at  the  end  of  this  report.  Syllabi  were 
rated  in  two  general  areas:  (1)  the  faculty  member  uses  and  models  it  and,  (2)  students  themselves  are  required  to 
use  it.  Faculty  use  of  a technology  tool  was  given  one  point  for  each  tool  and  student  use,  based  upon  constructivist 
educational  strategies,  was  given  two  points  per  tool.  In  addition  to  specific  tools  listed,  a category  entitled  “Other” 
allowed  for  the  tallying  of  technology  not  covered  by  the  survey.  The  use  of  word  processing  was  not  included  as  it 
already  plays  a prominent  role  in  the  university  classroom  (prior  to  any  technology  training  workshops).  Scores 
could  range  from  0 to  over  30  depending  upon  the  amount  of  “other”  uses  listed. 

Tallies  were  then  placed  in  an  ordinal  scale  that  ranked  the  use  from  “No  Use/Exposure  to  Technology”  to 
“Integrated  Use/Exposure”.  The  actual  scale  is  as  follows: 


Category 

Range 

No  Use/Exposure 

0 

Minimal  Use/Exposure 

1 - 7 

Adequate  Use/Expose 

8-  14 

Integrated  Use/Exposure 

15+ 

Categories  are  based  upon  the  amount  of  student  exposure  that  was  considered  reasonable  by  the  TIPT 
advisory  committee.  It  was  felt  that  a minimum  rating  of  15  indicated  quite  a bit  of  exposure  and  use  without 
compromising  course  integrity  due  to  overuse  of  technology  for  the  sake  of  technology  itself.  No  attempt  was  made, 
however,  at  this  time  to  determine  the  appropriateness  of  the  integration  of  technology.  This  issue  will  be  measured 
spring  2003. 


Findings 

Overall  ratings  ranged  from  1 to  21.  Mean  average  was  8.85  (adequate  use/exposure)  with  a standard 
deviation  of  4.97.  This  calculation  was  based  upon  the  frequency  of  tools  used  (interval  scale)  rather  than  the  ordinal 
categorization.  Frequencies  of  scores  in  the  ordinal  categories  are  displayed  in  the  graph  below: 

Use  of  Technology  in  the  Classroom 


CO 

0 

1 

No  Use  Minimal  Use  Adequate  Use  Integrated  Use 


Frequencies  indicate  that  12  or  60%  of  the  respondents  have  already  incorporated  the  use  of  technology 
into  their  course  instruction  at  an  adequate  level.  It  is  expected  that  continued  professional  development,  along  with 
the  sharing  of  ideas  and  strategies  among  faculty  members,  would  increase  this  percentage  to  at  least  75%. 

In  addition  to  examining  overall  scores,  sub-scores  in  both  the  faculty  and  student  use  categories  were 
inspected.  In  this  case,  one  point  was  assigned  on  both  the  faculty  use  and  the  student  use  in  order  to  compare  the 
amount  of  tools  that  were  incorporated  amongst  the  two  groups.  Nine  tools  were  listed  as  well  as  the  “Other” 
category  that  was  not  limited  in  number.  Faculty  use  ranged  from  1 tool  to  7 and  student  use  ranged  from  0 (3 
courses  had  this  rating)  to  7.  The  tools  most  commonly  used  by  faculty  were  email  (17)  and  the  use  of  the  Internet 
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for  research  (11).  Student  use  paralleled  faculty  use  with  16  courses  requiring  students  to  use  email  and  10  requiring 
students  to  research  on  the  Internet.  Full  results  are  listed  in  Table  1. 


Table  1:  Faculty  and  Student  Use  of  Technology 


Tool 

Faculty  Use 

Student  Use 

Email 

17 

16 

Internet  for  research 

11 

10 

Web  pages  and  courses 

6 

6 

Multimedia  devices 

5 

1 

Presentation  software 

5 

1 

Electronic  portfolios 

0 

0 

Other 

21 

22 

The  “Other”  category  included  the  following  tools: 

Use  of  listserves 
Video 

Bulletin  boards 
Library  databases 

Adaptive  technologies  for  visually  impaired 
Foreign  language  teaching  forums  and  labs 
Teleconferencing 

Studying  the  use  of  technology  in  teaching  languages 

Live  text  software 

Digital  archives 

Graphing  calculator 

Statistical  software 


Conclusions 

After  one  TIPT  workshop  series,  faculty  members  have  already  begun  integrating  technology  into  the 
university  classroom.  While  the  most  common  uses  are  still  the  more  traditional  tools  (email  and  internet),  many 
participants  have  reported  using  technology  in  creative  ways  to  enhance  the  teaching/learning  experience.  It  is 
recommended  that  future  workshops  place  more  emphasis  upon  and  assistance  in  the  application  of  other  technology 
tools  to  enhance  instruction  (when  appropriate).  A fine  line  has  to  be  drawn  between  using  technology  to  enhance 
instruction  and  using  technology  simply  for  its  own  sake.  Electronic  portfolios  were  not  reported  as  being  used. 
However,  the  College  of  Education’s  Technology  Task  Force  is  currently  working  on  developing  guidelines  for 
electronic  portfolios  and  their  integration  into  college  capstone  courses.  These  guidelines  should  be  available  spring 
2003. 

Findings  should  be  viewed  with  caution  as  the  rating  scale  was  determined  somewhat  arbitrarily  and  the 
quality  or  appropriateness  of  the  use  of  technology  was  not  recorded.  It  is  encouraging,  however,  that  many  faculty 
participants  are  already  making  use  of  the  knowledge  gained  through  the  TIPT  workshops.  This  first  assessment  will 
serve  as  a baseline  from  which  progress  can  be  gauged. 

TIPT  Infusion  of  Technology  into  the  Curriculum  Scoring  Rubric 

This  rubric  is  to  be  used  to  evaluate  course  syllabi.  When  the  syllabi  do  not  indicate  the  extent  to  which 
technology  has  been  incorporated  into  the  course,  faculty  comments  will  be  solicited. 

College/department  of  course: 


Technology  Tool 

Faculty  Uses  It  (1  pt.) 

Students  Use  It  (2  pts.) 

E-mail 

Internet  research 

Web-based  course 

Multi- media  (cameras,  etc.) 
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Teaching  tools  (grades  books,  etc.) 

Presentation  software 

E-portfolio 

Tutorials 

Concept  mapping 

Other  (list  below) 

Total  per  column: 

Total  score:_ 

Please  list  other  uses  not  mentioned  in  the  rubric: 
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Online  Learning  Students’  Profiles 


Rodrigo  del  Valle 
Thomas  M.  Duffy 
Indiana  University 

Introduction 

Interest  in  online  education  has  grown  rapidly  over  the  past  and  will  continue  to  grow  (Bonk,  2002;  Sofres, 
2001).  An  increasing  number  of  universities  are  offering  online  learning  (OL)  opportunities  (Duffy  & Kirkley, 
2004b).  Teacher  professional  development  is  part  of  this  trend  as  well.  Numerous  recent  studies  of  online  teacher 
professional  development  environments  have  been  published.  Most  renowned  are  the  cases  of  Tapped  In  (Schlager, 
2004),  ILF  (Barab,  MaKinster,  & Scheckler,  2004),  LTTS  (Duffy,  Kirkley,  del  Valle,  & Malopinsky,  2004),  and  the 
Math  Forum  (Renninger  & Shumar,  2004). 

As  Duffy  & Kirkley  (2004a)  point  out,  while  many  view  OL  as  a highly  promising  educational  opportunity, 
offering  several  advantages  including  not  only  access  aspects,  but  also  reforming  teaching  practices  and  offering 
new  models  for  life  long  learning,  others  view  OL  as  a threat  to  education  that  lower  the  quality  of  instruction.  Like 
in  traditional  face-to-face  learning,  OL  design  and  implementation  vary  widely,  and  so  do  instructional  quality  and 
levels  of  students’  engagement  and  learning.  OL  might  simply  repeat  the  problems  of  traditional  professional 
development,  but  it  can  be  used  to  implement  alternative  professional  development  approaches  while  meeting  the 
practical  needs  of  teachers. 

Multiple  studies  have  compared  online  versus  face-to-face  learning,  with  some  results  showing  positive 
outcomes  for  OL  (Hiltz,  Coppola,  Rotter,  & Turoff,  2000;  Olson,  2002),  and  others  yielding  non-significant  results 
(Russell,  1999)  showing  that  web-based  can  be  as  effective  as  campus-based  environments  (Hall,  1999).  But  more 
important  than  media  comparisons  are  the  instructional  approaches  used  (Clark,  1983,  1994),  the  variables  that 
might  affect  learning  in  each  environment,  and  the  affordances  and  constraints  they  have.  These  factors,  along  with 
quality  implementation  and  design,  determine  how  effective  a learning  approach  can  be. 

In  general,  we  know  that  what  students  “do”  or  do  not  do  is  critical  to  their  learning.  We  expect  students  to 
read,  reflect,  write,  discuss,  etc.  From  a constructivist  perspective,  the  learner’s  goals  affect  what  they  do  and 
therefore  what  is  learned  and  how  it  is  learned  (Duffy  & Cunningham,  1996;  Duffy  & Orrill,  2004;  Jonnasen,  1999). 
If  a teacher  is  taking  a course  mainly  to  renew  her  license,  she  will  learn  differently  than  if  she  is  working  on  the 
course  to  apply  what  she  learns  in  her  classroom.  The  understandings  she  is  constructing  will  be  different,  and  so 
will  her  approach  to  the  course  activities,  and  the  amount  of  energy  devoted  to  them. 

One  of  the  seeming  advantages  students  perceive  in  asynchronous  courses  is  having  the  freedom  and 
flexibility  in  organizing  their  learning  activities.  They  have  the  ability  to  make  decisions  about  what  resources  or 
tools  to  use,  when,  in  what  order,  or  for  how  long.  Flexibility  and  learner  freedom  are  attributes  that  are  particularly 
present  in  online  open-ended,  learner-centered  environments.  The  possibilities  for  sustained  discourse  and 
discussion  using  asynchronous  tools,  in  which  learners  have  time  to  think,  and  contributions  are  recorded,  have  great 
potential  for  promoting  critical  thinking.  OL  can  also  support  the  instructor’s  ability  to  provide  just  in  time  feedback, 
and  develop  one-on-one  mentoring  strategies.  Easy  access  to  a variety  of  learning  resources,  and  the  possibilities  for 
supporting  communities  of  practice,  also  illustrate  the  enormous  potential  OL  has  (Barab,  Kling,  & Gray,  2004; 

Bonk  & King,  1998;  Duffy,  Dueber,  & Hawley,  1998;  Duffy  & Kirkley,  2004b). 

We  began  the  previous  paragraph  noting  the  “seeming”  advantage  of  the  asynchronous,  online 
environments;  while  many  students  flourish  with  the  freedom,  many  others  flounder.  High  dropout  rates  in  distance 
education  reflect  this  situation  (Bonk,  2002;  Parker,  1999).  Thus,  as  asynchronous  learning  environments  flourish,  it 
is  important  to  begin  to  understand  OL  strategies.  To  understand  how  learners  work  online,  how  they  interact  with 
the  facilitator,  how  they  use  available  resources,  and  how  those  strategies  relate  to  learning  are  salient  questions  for 
both  course  design  and  mentoring  strategies. 

Unfortunately,  we  do  not  know  much  about  how  learners  work  online.  In  traditional  face-to-face  contexts, 
it  is  usually  possible  to  observe  learners.  But  in  OL  environments,  especially  asynchronous  ones,  this  is  not  feasible: 
since  we  cannot  observe  them,  we  do  not  know  what  students  are  “doing,”  or  what  their  reactions,  attitudes,  and 
levels  of  engagement  are.  In  OL  we  can  get  a good  sense  of  what  students  “click  on,”  but  it  is  harder  to  know  what 
they  do  after  the  click.  Nonetheless,  OL  courses  can  offer  information  about  what  learners  are  “doing”  at  home 
when  they  are  working  online,  which  is  not  possible  in  face-to-face  environments.  Complex  log-files  (click-stream 
data),  along  with  other  forms  of  data,  allow  us  to  paint  a rich  description  of  what  learners  do  while  working  online. 
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Click-stream  data  is  not  limited  to  counting  the  number  of  clicks.  Structural  measures  (patterns  of  activity),  and 
temporal  measures  (when  and  for  how  long)  can  complement  basic  “click’'  data  to  help  us  follow  and  understand  the 
“footprints”  learners  leave  when  working  online. 

There  have  been  several  studies  in  recent  years  examining  how  learners  use  hypermedia  and  web  based 
environments.  Most  of  these  studies  use  log  files  as  an  efficient  and  non-intrusive  way  of  studying  and 
understanding  the  dynamic  nature  of  web  based  learning  (Hall,  Balestra,  & Davis,  2000;  Young  & McNeese,  1995). 

Barab,  Young,  & Wang  (1999),  and  Hall  et  al  (2000)  focused  on  the  impact  of  different  learning  interfaces 
and  learner  control,  others  focused  on  individual  differences  and  students  profiles  (Barab,  Bowdish,  & Lawless, 

1997;  Barab,  Fajen,  Kulikowich,  & Young,  1996;  Ford  & Chen,  2000;  Lawless  & Kulikowich,  1996,  1998). 

Barab,  et  al.  (1997),  for  example,  presented  students  with  a campus  Kiosk  and  gave  students  tasks  of 
finding  information.  Using  cluster  analysis  to  analyze  click  stream  data  of  the  types  of  pages  examined  and  the  depth 
of  those  pages  they  were  able  to  identify  4 types  of  users  according  to  the  way  they  navigate  on  the  kiosk: 

■ Model  users:  compliant  and  earnest,  pick  the  simplest  task,  fewer  deviations 

■ Disenchanted  volunteers:  rebellious  and  impatient,  explored  very  little 

■ Feature  explorers:  featured  oriented  and  confused,  use  help  screens,  lowest  self-efficacy 

■ Cyber  cartographers:  curious,  goal  directed,  longer  time,  deepest  levels,  highest  self-efficacy 

Some  studies  analyzed  student’s  hypermedia  traversal  focusing  on  the  number  of  pages  they  visit  (on 
different  categories),  others  on  the  pattern  they  use.  Hall  et  al.  (2000)  considered  both  approaches  using  a cardinal 
measure  (numbers  of  times  a page  is  visited),  a structural  measure  (pattern  of  linking),  and  adding  also  a temporal 
one  (time  on  pages).  With  this  comprehensive  model  for  the  analysis  of  hypermedia  navigation.  Hall  et  al.  analyzed 
student’s  use  of  a traditional  linear  interface  versus  a non-liner  “hypermap”  interface.  The  results  indicate  that  the 
two  groups  were  similar  on  the  cardinal  navigation  measure,  but  the  temporal  and  structural  measures  clearly 
differed,  showing  that  a comprehensive  analysis  of  navigation  patterns  can  provide  useful  insight  to  understand 
hypermedia  processing. 

Lawless  & Kulikowich  (1998)  emphasize  the  importance  of  cognitive  and  affective  variables,  especially 
domain  knowledge,  individual  interest  and  situational  interest,  with  regard  to  different  profiles.  Their  research  lead 
to  the  following  profiles:  Knowledge  seekers,  Feature  Explorers  and  Apathetic  hypertext  users  (Lawless  & 
Kulikowich,  1996). 

It  is  important  to  notice  that  it  only  makes  sense  to  study  online  learning  profiles  within  a flexible 
environment  in  which  learners  have  the  freedom  to  make  enough  decisions  about  their  work  so  clearly 
distinguishable  approaches  may  emerge.  A fixed  linear  environment  with  a strictly  defined  set  of  required  readings 
and  activities  would  most  probably  leave  very  little  space  for  different  learning  profiles  to  emerge. 

The  studies  we  have  reviewed  offered  a flexible  environment,  but  they  were  conducted  mainly  over  brief 
experiences,  most  of  them  in  experimental  context,  in  which  the  learner  interacted  with  the  environment  over  a 
relatively  short  period  of  time  with  an  instructional  task  assigned  by  the  researchers.  While  those  studies  provide  a 
context  for  the  present  research,  this  one  focuses  on  authentic  and  more  prolonged  learning  environments;  real  in- 
service  and  pre -service  teachers  taking  online  professional  development  one-credit  courses. 

Method 

Learning  Environment 

The  freedom  learners  have  while  working  in  online  learner-centered-environments  is  especially  prevalent 
in  the  Learning  to  Teach  with  Technology  Studio  professional  development  environment  (LTTS: 
http://ltts.indiana.edu).  LTTS  consists  of  a catalogue  of  50  facilitated  self-paced  web-based  courses.  The  increased 
freedom  of  the  self-paced  environment  provides  the  students  with  even  greater  control  over  their  learning  practices, 
permitting  them  to  use  the  approach  that  is  comfortable  in  their  learning.  They  can  plow  through  systematically, 
jump  around,  procrastinate  and  then  rush  through  it,  be  thoughtful  and  systematic,  etc.  Considering  these 
characteristics,  LTTS  courses  were  used  as  the  study  learning  environment.  Courses  are  short  (between  20  and  25 
hrs),  entirely  web  based,  self-paced,  individually  mentored,  and  address  technology  integration  on  learner  centered 
teaching.  Course  design  is  guided  problem  solving  (Malopinsky,  2000;  Savery  & Duffy,  1996),  providing  a non- 
linear, resource-rich  environment  with  an  open-ended  structure.  Four  to  seven  tasks  guide  student  work,  and  the 
outcome  is  a student -designed  product  to  use  in  his/her  professional  work. 

Participants  and  Procedure 

The  participants  were  graduate  students  enrolled  at  a large  Midwestern  university  in  the  graduate  course 
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“Elementary  and  Secondary  School  Curriculum”.  Twenty  of  twenty-three  students  selected  LTTS  as  their  option  to 
fulfill  the  technology  integration  requirement  in  the  course,  and  all  twenty  agreed  to  participate  in  the  study. 

Fifteen  of  the  twenty  participants  were  between  the  ages  of  20  and  30,  while  five  were  over  40.  Eight  of  the 
participants  had  been  or  were  full  time  teachers  returning  to  school,  while  the  other  twelve  had  only  field  work  and 
student  teaching  experience.  Of  those  with  no  full  time  teaching  experience,  one  was  a technology  coordinator  and 
one  was  a library  specialist.  A wide  range  of  subject  areas  and  grade  levels  were  represented  and  this  is  reflected  in 
the  fact  that  thirteen  different  LTTS  courses  were  selected  by  the  twenty  participants.  All  learners  were  randomly 
assigned  to  one  of  two  LTTS  trained  facilitators. 

Click  stream  data,  including  cardinal,  structural,  and  temporal  measures,  were  collected  for  all  actions 
learners  made  while  online.  Table  one  describes  the  eight  critical  variables  that  were  defined  in  order  to  determine 
student  profiles  on  LTTS  self-paced  online  courses.  Although  the  amount  of  click  stream  data  LTTS  collects  would 
allow  us  to  define  dozens  of  variables,  hierarchical  cluster  analysis  requires  that  only  a limited  set  of  meaningful  and 
distinct  variables  is  selected.  In  this  study  only  those  variables  that  were  meaningfully  related  to  student’s  online 
freedom  and  flexibility  in  the  LTTS  environment  were  selected. 


Table  1:  List  of  variables  and  their  scales  and  definitions 


Variable 

Definition 

Sessions 

Number  of  sessions  used  to  complete  the  course.  A session  is  defined 
as  a series  of  activities  not  more  than  15  minutes  apart,  so  after  a user  is 
inactive  for  more  than  15  minutes  the  session  is  consider  being  over 

Days 

Number  of  calendar  days  from  beginning  to  completion  of  the  course 

Online  Time 

Total  number  of  hours  spent  online 

Interval  Mean 

Average  number  of  days  between  logins 

Interval  Standard  Deviation 

Standard  deviation  of  number  of  days  between  logins 

Time  on  Resources 

Proportion  of  time  spent  on  content  resources  provided  on  the  course 

Messages  Read 

Number  of  facilitator  messages  read.  Facilitator  messages  consist 
mainly  of  feedback  for  course  activities,  and  they  are  critical  to  the 
LTTS  pedagogical  design.  Students  post  their  work  on  a workbook, 
and  their  messages  outside  this  environment  are  reduced  to  a minimum, 
therefore  they  were  not  considered.  Facilitator  messages  are  often  long 
and  rich  in  content  so  students  often  go  back  to  them.  Thus  this 
variable  considered  each  time  a message  was  read. 

Transitions 

Number  of  times  the  learner  “jumped”  between  course  activities  not 
following  a linear  path.  Following  a total  linear  path  through  the  course 
activities  would  imply  cero  transitions.  More  transitions  indicate  going 
back  and  forward  through  course  activities. 

Results  and  Analysis 


Cluster  Analysis 

Click  stream  data  were  collected  for  all  20  participants  until  they  completed  the  course.  Following  Barab  et 
al  (1997)  Ward’s  (1963)  hierarchical  cluster  analysis  was  used  to  identify  naturally  occurring  groups  of  students.  A 
scree-plot  (Figure  1)  was  used  to  evaluate  the  between  clusters  distances,  and  determine  the  number  of  meaningful 
clusters  according  to  the  larger  between  clusters  variability.  When  the  coefficient  drops  below  7500,  between 
clusters  16  and  17,  the  scree-plot  levels  off  indicating  that  the  variability  between  clusters  stops  suggesting  the 
existence  of  separate  groups.  Therefore,  the  data  supports  the  existence  of  three  clearly  differentiated  clusters  or 
groups  of  learners  with  similar  approaches  to  online  learning  according  to  the  defined  set  of  variables. 
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Figure  1:  Scree-plot  used  to  determine  number  of  meaningful  clusters 


Cluster  Pairs 


Mean  scores  and  standard  deviations  for  each  dependent  variable  on  all  three  clusters  (Table  2)  show  how 
clusters  comparatively  behave  on  each  variable.  Refer  to  table  1 for  a detailed  description  of  the  variables  and  their 
scales. 


Table  2:  Mean  scores  and  standard  deviations  for  each  dependent  variable  separated  by  cluster 


Cluster  1 
Task  Oriented 
Learners  (n=3) 

Cluster  2 

Grade  Oriented  Learners 
(n=14) 

Cluster  3 
Goal  Oriented 
Learners  (n=3) 

Variable 

M 

SD 

M 

SD 

M 

SD 

Sessions 

22.33 

6.66 

37.21 

10.21 

45.00 

14.11 

Days 

23.67 

8.39 

60.79 

6.65 

31.33 

7.77 

Online  Time 

5.17 

3.30 

6.22 

2.28 

10.54 

4.42 

Interval  Mean 

1.24 

0.67 

1.84 

0.73 

0.76 

0.29 

Interval  SD  Mean 

2.84 

1.81 

3.32 

1.72 

1.09 

0.55 

Time  on  Resources 

19.17 

15.48 

17.86 

9.38 

26.03 

13.71 

Messages  Read 

19.00 

3.61 

34.43 

8.11 

49.67 

2.31 

Transitions 

2.67 

2.08 

16.21 

8.50 

57.00 

24.25 

Standardized  means  were  used  to  compare  clusters  scores  in  a complex  set  of  variables  that  had  different 
scales  and  a wide  variety  of  ranges.  A graphical  depiction  of  the  Z scores  for  the  eight  studied  variables  (Figure  2) 
illustrates  how  learners  on  each  of  the  clusters  comparatively  behaved  on  each  of  the  variables  that  shaped  the 
profiles  in  a unique  way. 
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Figure  2:  Depiction  of  the  standardized  means  of  the  3 clusters  on  the  studied  variables 
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Individuals  on  the  first  cluster  (n=3)  present  in  average  the  lowest  number  of  course  sessions  (M=22, 

SD=6.66),  the  lowest  online  time  (M=5.17  hours,  SD=3.30)  and  the  lowest  number  of  days  invested  in  the  course 
(M=23.67,  SD=8.39).  According  to  these  results,  they  tend  to  spend  only  the  indispensable  time  on  the  course 
without  spreading  it  over  several  days  or  sessions.  They  tend  to  be  focused  on  their  work,  completing  the  course  as 
soon  as  possible  with  an  average  of  just  over  3 weeks.  At  the  same  time,  they  have  the  lowest  number  of  transitions 
(M=2.67,  SD=2.08),  indicating  a linear  path  trough  the  course  activities  with  minimal  deviations,  revisions  of 
previous  work  or  exploration  of  future  activities.  They  also  have  the  lowest  number  of  facilitator  messages  read 
indicating  less  contact  with  their  facilitators,  and  minor  reviewing  of  feedback  messages.  Regarding  the  use  of 
resources,  they  spent  about  20%  of  their  online  time  reviewing  the  content  resources  provided  on  the  course. 
Considering  these  characteristics,  we  describe  this  group  as  “task  oriented  learners”;  they  have  an  assignment,  and 
they  want  to  complete  it  efficiently  and  quickly. 

Grade  oriented  learners 

The  second  cluster  is  the  largest  one  (n=14).  Members  of  this  cluster  used  the  highest  number  of  days  to 
complete  the  course  (M=60.79,  SD=6.65),  and  their  work  intervals  are  the  most  extended  ones  with  an  average  of 
almost  two  days  between  logins  (M=1.84,  SD=0.73).  They  tend  to  spread  their  work  over  time,  with  not  very 
frequent  logins,  and  on  average  took  them  almost  nine  weeks  to  complete  a course.  Nevertheless,  despite  of  all  days 
spent  on  the  course  they  have  only  an  intermediate  total  online  time  (M=6.22,  SD=2.28),  intermediate  number  of 
messages  read,  intermediate  number  of  transitions  (wander  around  the  course  and  review  some  activities),  and  the 
lowest  time  on  resources.  As  a result,  we  called  this  group  “grade  oriented  learners”.  They  seem  to  have  no  hurry  to 
complete,  and  want  to  earn  the  course  credit  spending  only  a minimum  time  on  the  available  resources,  with  an 
intermediate  level  of  commitment. 

Goal  oriented  learners 

The  third  cluster  (n=3)  represents  those  students  with  the  highest  number  of  course  sessions  (M=45.00, 
SD=14.1 1),  and  the  highest  online  time  with  an  average  of  10.54  hours  (SD=  4.42).  They  also  have  the  lowest  work 
intervals,  and  an  intermediate  number  of  days  spent  on  the  course  with  an  average  of  31.33  days  (SD=7.77).  They 
work  very  intensively  on  the  course,  and  logged  in  very  often  spending  an  average  of  about  4 weeks  to  complete  the 
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course.  They  also  have  the  highest  number  of  transitions,  the  highest  time  on  resources,  and  the  highest  number  of 
messages  read.  This  indicates  that  they  tend  to  revisit  their  course  work,  review  previous  and  subsequent  activities, 
and  ponder  facilitator  feedback.  Taking  into  account  these  characteristics,  we  have  labeled  this  profile  as  “goal 
oriented  learners”  since  they  seem  to  be  highly  committed  to  the  course  and  self-driven  in  their  work. 

Discussion 

The  present  study  used  click  stream  data,  including  cardinal,  structural,  and  temporal  measures,  to  identify 
online  student’s  profiles.  Coinciding  with  the  informally  reported  perceptions  that  LTTS  facilitators  have  about  the 
different  approaches  learners  take  when  working  on  the  courses,  cluster  analysis  led  to  the  identification  of  three 
clearly  differentiated  groups  of  learners:  task  oriented,  grade  oriented,  and  goal  oriented  learners. 

The  sole  existence  of  these  three  groups  confirms  that  different  learners  make  a different  use  of  the  freedom 
and  flexibility  provided  by  the  self-paced  environment.  For  example,  while  some  learners  (goal  oriented)  tend  to 
“jump  around”  the  course  having  a high  number  of  transitions,  others  followed  a more  linear  path. 

Regarding  the  use  of  time,  learners  do  make  use  of  the  flexibility  provided  by  the  self-paced  environment 
having  three  clearly  differentiated  groups.  Nevertheless,  those  that  used  more  days  to  complete  the  course  (grade 
oriented)  were  not  the  ones  that  spent  more  online  time  or  had  more  courses  sessions.  Similarly,  different  learners 
spent  different  amount  of  time  using  the  course  resources,  this  being  in  some  cases  over  40%  of  their  total  online 
time,  and  in  others  less  than  10%.  In  this  sense,  it  seems  relevant  to  provide  enough  resources  for  all  types  of 
learners,  and  the  flexibility  to  use  only  part  of  them. 

Even  if  students’  approaches  to  OL  are  not  substantially  different  from  those  of  face-to-face  courses,  online 
environments,  and  the  reasons  why  learners  take  online  courses,  might  exacerbate  the  characteristics  of  different 
profiles,  or  perhaps  online  courses  augment  a self  selection  of  certain  learning  approaches.  Given  how  online 
learning  might  meet  the  personal  goals  and  practical  needs  teachers  have  when  pursuing  professional  development, 
we  could  have  expected  that  the  task  and  goal  oriented  profiles  would  amount  for  a larger  number  of  students,  but 
those  two  profiles  account  for  only  30%  of  the  participants.  Nevertheless,  it  is  important  to  notice  that  although 
participation  in  the  study  was  voluntary,  participants  choose  to  take  an  LTTS  course  in  order  to  fulfill  a requirement 
of  the  graduate  course  they  were  enrolled  in.  This  situation  may  have  affected  the  predominance  of  students  in  the 
grade  oriented  learner  profile  (70%). 

The  results  of  this  study  might  have  important  implications  for  the  type  of  support  mentors  have  to  provide 
to  different  students.  For  example,  while  task-oriented  learners  seem  to  rest  less  on  facilitator  feedback,  grade- 
oriented  and  goal  oriented-learners  tend  to  interact  more  with  facilitators.  In  the  same  way,  facilitators  can  expect  to 
have  some  learners  that  will  promptly  complete  the  course  (task-oriented)  while  grade -oriented  learners  will 
probably  need  frequent  reminders  in  order  to  pursue  their  work.  If  this  is  the  case,  future  studies  should  also 
investigate  how  mentoring  strategies  in  online  learner-centered  environments  (Collison,  Elbaum,  Haavind,  & 

Tinker,  2000)  relate  to  students’  OL  approaches. 

Since  this  study  was  conducted  with  participants  that  completed  the  online  course  they  were  taking,  one 
important  group  that  obviously  could  not  be  analyzed  was  the  students  who  dropout  of  OL.  Reasons  for  learner 
attrition  in  general  (Tinto,  1987),  and  in  OL  in  particular  (Beatty,  Malopinsky,  & Duffy,  2003)  have  been  studied, 
yet  we  need  to  investigate  what  can  be  done,  from  a design  and  facilitation  perspective,  in  order  to  support  teachers 
and  other  professionals  seeking  online  professional  development,  and  to  reduce  their  attrition  rates.  Taking  into 
account  the  existence  of  different  OL  profiles  might  prove  useful  in  these  efforts. 

Although  LTTS  presents  a rich,  unique,  and  successful  online  teacher  professional  development 
environment  (Duffy  et  ah,  2004),  using  only  one  learning  environment  is  a limitation  of  this  study.  The  possibility  of 
replicating  these  efforts  using  other  learner-centered  online  environments  should  be  explored. 

Unfortunately,  this  study  was  conducted  using  a small  sample.  Nevertheless,  current  results  call  for 
promising  replications,  and  more  studies,  with  larger  samples,  linking  these  profiles  to  other  variables  affecting  OL 
are  needed.  In  addition,  collecting  qualitative  data  would  be  helpful  for  interpreting  results,  and  for  triangulation  to 
support  or  dispute  findings. 

Linking  OL  profiles  with  variables  affecting  OL  strategies,  including  self-efficacy,  trust,  learning  goals, 
beliefs  about  learning,  and  previous  experience  would  provide  a rich  context  to  study  online  learning  strategies.  In 
addition,  studying  the  interaction  between  different  learner  profiles  and  outcome  variables  such  as  learner 
satisfaction,  perceived  and  actual  learning,  and  transfer  expectations,  will  also  be  relevant  for  the  future  of  OL, 
especially  in  the  context  of  teacher  professional  development. 

This  study,  and  the  results  of  other  similar  studies  we  are  currently  conducting,  might  yield  interesting 
findings  that  will  help  the  OL  community  understand  effective  online  learning  and  facilitation  strategies  for  adult 
learners  (teachers).  Findings  could  have  implications  for  the  type  of  support  facilitators  must  provide  for  learners 
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with  different  profiles,  and  for  how  online  courses  are  designed  and  implemented.  Finally,  if  one  approach  were 

found  to  be  more  successful  than  others,  then  investigating  how  to  foster  that  approach  using  specific  facilitation 

strategies  and  course  instructional  design,  would  also  be  relevant. 
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Interest  in  online  learning  (OL)  has  grown  rapidly  and  will  continue  to  do  so  (Bonk,  2002;  Sofres,  2001). 

An  increasing  number  of  universities  are  offering  OL  opportunities  (Duffy  & Kirkley,  2004).  A number  of  studies 
have  discussed  and  linked  online  collaboration  through  group  discussions  with  better  opportunities  to  promote 
quantity  and  quality  of  student  interaction,  engagement,  satisfaction,  and  higher-order  learning  (Hiltz,  Coppola, 
Rotter,  & Turoff,  2000;  Mikulecky,  1998;  Nachmias  et  al.,  2000).  Nevertheless,  it  is  common  to  see  little  discussion 
among  online  groups  that  work  collaboratively.  Participants  tend  to  post  without  replying  to  other  participants’ 
contributions,  and  they  remain  in  their  comfort  zones  (Collett,  Kanuka,  Blanchette,  & Goodale,  1999).  Threaded 
discussions  often  end  in  trivialized  conversations  in  which  frequently  several  members  remain  as  passive  observers 
(Klenrm  & Snell,  1996).  As  Kanuka  & Garrison  (2004)  point  out  “To  date  there  have  been  few  empirical  studies  on 
the  use  of  asynchronous  text -based  Internet  communication  technologies  and  their  ability  and/or  effectiveness  to 
facilitate  higher  levels  of  learning’’  (p.31).  OL  might  have  great  potential  to  promote  critical  thinking,  however,  to 
implement  OL,  and  especially  online  discussions  in  a way  that  actualize  this  potential  has  proved  to  be  a real 
challenge. 

One  of  the  main  arguments  in  studies  that  have  linked  online  collaboration  through  group  discussions  with 
better  learning  opportunities  is  that  group  discourse  serves  as  a core  in  increasing  individual  critical  thinking  within 
OL  environments  (Angeli  & Bonk,  2003;  Garrison,  Anderson,  & Archer,  2001;  Jeong,  2001).  Critical  thinking  is 
defined  as  “the  intellectually  disciplined  process  of  actively  and  skillfully  conceptualizing,  applying,  analyzing, 
synthesizing,  and/or  evaluating  information  gathered  from,  or  generated  by,  observation,  experience,  reflection, 
reasoning  or  communication,  as  a guide  to  belief  and  action  (Paul  & Elder,  2001).’’  Duffy,  Dueber,  & Hawley 
(1998)  place  inquiry  at  the  heart  of  learner-centered  online  environments,  and  associate  critical  thinking  with  an 
inquiry  process  in  which  a learner  inductively  or  abductively  solves  his  puzzlement  through  hypothesis  generation, 
data  gathering  and  evaluation,  considering  alternatives,  and  resolving  for  a rational  solution. 

While  critical  thinking  tends  to  be  defined  as  an  individual  cognitive  process,  Garrison,  Anderson,  & 

Archer  (2000)  developed  a “community  of  inquiry’’  model  that  provides  a conceptual  framework  to  study  learning  in 
online  environments,  placing  critical  thinking  in  the  context  of  collaborative  work.  In  this  model  they  use  the  term 
“cognitive  presence’’  to  refer  to  “the  extent  to  which  learners  are  able  to  construct  and  confirm  meaning  through 
sustained  discourse  in  a critical  community  of  inquiry  (Garrison  et  al.,  2001),”  focusing  in  this  way  on  the  sustained 
interactions  that  help  individuals  to  develop  critical  thinking  as  a process  and  outcome  (Archer,  Garrison,  Anderson, 
& Rourke,  2001;  Garrison  et  al.,  2000). 

In  addition  to  cognitive  presence,  Garrison  et  al.  (2000)  also  emphasize  the  importance  of  social  presence 
in  their  community  on  inquiry  model.  Social  presence  is  the  sense  of  relationship  that  can  arise  among  people 
participating  in  online  environments,  because  participants  project  their  personal  characteristics  into  the  community 
of  inquiry,  thereby  presenting  themselves  as  real  people  (Rourke,  Anderson,  Garrison,  & Archer,  2001).  In  OL 
social  presence  can  be  mediated  using  text  based  communication,  and  it  appears  to  be  relevant  to  promote  student 
satisfaction  and  engagement  (Richardson  & Swan,  2003).  Nonetheless,  Wise,  Chang,  Duffy,  &del  Valle  (2004) 
found  that  only  a basic  threshold  of  social  presence  is  required  in  orderto  support  online  learners. 

The  role  of  online  facilitators  who  cultivate  learners’  cognitive  presence  has  been  considered  as  an 
important  factor  in  promoting  students’  critical  thinking  (Anderson,  Rourke,  Garrison,  & Archer,  2001;  Archer  et 
al.,  2001;  Collison,  Elbaum,  Haavind,  & Tinker,  2000;  Richardson  & Swan,  2003).  Thus,  identifying  guidelines  for 
facilitating  cognitive  presence  in  online  settings  is  an  essential  issue  in  order  to  encourage  learners  to  think  critically 
in  a collaborative  environment.  Collison  et  al.  (2000)  identified  several  facilitation  strategies  to  promote  critical 
thinking  in  online  environments.  These  strategies  suggest  that  facilitators  should  promote  cognitive  presence  in  two 
stages:  in  the  first  stage,  the  facilitator  uses  techniques  to  help  learners  clarify  their  ideas,  and  come  to  a common 
understanding  in  order  to  “sharpen  the  focus”.  In  the  second  stage,  the  facilitator  supports  learners  to  deepen  their 
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thoughts  and  dialogue  by  articulating  more  in-depth  discussions  on  the  common  ground.  In  this  context  the 
facilitator  is  not  acting  as  a content  expert  but  as  an  experienced  learner  and  mentor  that  promotes  students  critical 
thinking  and  knowledge  construction,  mainly  by  asking  questions  that  will  further  push  their  thinking  and  inquiry. 

The  purpose  of  this  exploratory  experimental  study  was  to  examine  how  online  cognitive  facilitation  that 
promotes  cognitive  presence,  while  keeping  facilitator’s  social  presence  constant,  affects  student  learning  and 
satisfaction  in  an  online  community  of  inquiry.  In  this  context,  we  explored  the  following  questions: 

1)  Does  online  high  cognitive  facilitation  promote  collaboration  and  critical  thinking? 

2)  Does  online  high  cognitive  facilitation  promote  student  learning  and  improve  satisfaction? 

Methodology 

Participants 

Participants  were  12  graduate  students  studying  in  the  Instructional  Technology  Department  of  a large 
Midwestern  university,  with  1 1 of  them  enrolled  in  the  distance  education  (DEJ  masters  program,  and  one  of  them  a 
residential  student.  Participants  were  equally  distributed  in  terms  of  gender  (50%  male,  50%female).  Two 
participants  were  in  the  age  range  20-30,  three  in  the  range  31-40,  five  in  the  age  range  41-50,  and  two  of  them  were 
over  5 1 . 

The  participants  were  randomly  assigned  to  four  3-person  groups.  Three  of  the  participants  dropped  from 
the  study  after  the  first  days  of  discussion,  which  caused  3 groups  to  continue  as  2-person  groups.  Additionally,  two 
participants  who  were  active  during  the  discussion,  for  unknown  reasons  did  not  complete  the  online  final 
instruments. 

Task 

During  the  study,  students  were  involved  in  an  1 1-day  online  asynchronous  discussion  about  the  use  of  DE 
in  K-12  environments.  The  whole  discussion  period  lasted  13  days,  but  there  was  a 2-days  break  within  that  period 
due  to  the  Thanksgiving  holiday.  To  facilitate  the  discussion  each  group  was  asked  to  identify  the  five  most 
important  issues  that  should  be  considered  in  determining  the  role  of  DE  in  K- 12  education.  They  were  to  think  of 
their  team  as  consulting  to  state  departments  of  education  on  the  use  of  DE.  Participants  were  asked  to  post  the 
issues  they  agreed  on  and  the  rationale  for  each  of  them  by  the  end  of  the  discussion.  Four  brief  magazine  articles  on 
the  K-12  DE  topic  were  selected  to  serve  as  resources  for  the  discussion.  One  of  these  articles  was  given  to  all 
members  in  each  of  the  groups.  The  other  three  articles  were  distributed  randomly  among  the  members  of  each 
group  in  order  to  minimize  the  workload  of  the  participants,  and  to  foster  the  diversity  of  ideas  in  the  discussion. 

The  participants  were  instmcted  not  to  look  for  more  articles  on  the  topic,  not  to  share  articles  with  their  teammates, 
and  not  to  discuss  the  topic  with  other  members  of  their  class.  They  were  to  make  at  least  one  posting  everyday,  and 
avoid  using  email  except  in  emergency  or  to  communicate  personal  information  to  the  facilitator. 

Design 

The  independent  variable  was  the  degree  of  cognitive  facilitation  provided  by  the  facilitator,  with  two 
groups  receiving  high  cognitive  facilitation  and  two  groups  receiving  low  cognitive  facilitation. 

There  were  two  facilitation  teams  each  of  which  consisted  of  three  people  from  the  research  team.  Each 
team  was  responsible  for  the  facilitation  of  one  group  in  the  high  cognitive  facilitation  condition,  and  another  group 
in  the  low  cognitive  facilitation  condition.  One  person  from  each  team  took  the  lead  in  generating  and  posting 
messages , and  the  others  gave  feedback  before  the  messages  were  posted.  Facilitators  were  to  post  a message  every 
day. 

In  both  high  and  low  conditions,  facilitators  avoided  using  didactic  teaching  strategies,  acting  as  content 
experts,  and  introducing  new  material.  Both  facilitators  included  high  social  presence  in  their  postings.  For  the 
purpose  of  this  study  facilitator  high  social  presence  is  defined  as  a style  of  communication  that  makes  active  use  of 
written  expressions  to  compensate  for  the  lost  non-verbal  cues  in  computer  mediated  communication  (Rice  & Love, 
1987).  The  intentional  use  of  social  presence  cues  (Swan,  2003;  Rourke,  Anderson,  Garrison,  & Archer,  2001; 
Abdullah,  1999)  should  allow  facilitators  to  be  perceived  as  friendly  within  the  mediated  communication.  The  cues 
used  were  humor/playful  asides,  expression  of  emotion,  self -disclosure,  addressing  people  by  name, 
greetings/phatics,  and  allusions  of  physical  presence. 

In  the  high  cognitive  facilitation  condition  the  overall  goal  of  the  facilitators  was  to  promote  critical 
dialogue  and  evidence  based  argumentation  among  team  members  without  providing  direction  or  points  of  view  on 
the  specific  content.  Cognitive  facilitation  included  inquiry  learning  facilitation  techniques  such  as  asking  for  the 
clarification  and  elaboration  of  the  presented  ideas,  promotion  of  the  use  of  the  articles,  and  the  sorting  and  synthesis 
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of  ideas.  In  the  low  cognitive  facilitation  condition  facilitators  did  not  use  any  of  these  strategies  but  rather 
encouraged  the  students  to  discuss  and  share  their  ideas. 

Facilitators  were  trained  in  responding  with  high  and  low  cognitive  facilitation  messages.  Throughout  the 
facilitation  process  they  used  two  facilitation  guides  prepared  by  the  research  team;  one  for  each  condition.  The  high 
cognitive  facilitation  guide  (Table  1)  included  specific  strategies  and  detailed  examples  for  each  one.  The  low 
cognitive  facilitation  guide  focused  mainly  on  social  and  motivational  presence,  and  instructed  facilitators  to  avoid 
using  any  of  the  strategies  used  on  the  high  cognitive  facilitation.  The  strategies  on  the  high  cognitive  facilitation 
guide  (Table  1)  were  drawn  from  the  model  defined  by  Collison  et  al.  (2000). 

Table  1:  Cognitive  Facilitation  Strategies*  (adapted from  Collison  et  al.,  2000) 

General  Strategy  Category  Specific  Strategies 

IDENTIFYING  DIRECTION:  Selectively  highlight  or  paraphrase  relevant 
discussion  items  in  order  to  refocus  or  redirect  conversation,  possibly 
weaving  several  discussion  threads  or  ideas  to  provide  a new  focus. 

SORTING  IDEAS  FOR  RELEVANCE:  Ask  students  to  classify  or  form 
comparisons.  Alternatively,  if  needed  you  can  call  attention  to  sorting  of 
ideas,  making  public  the  sorting  mechanism,  to  focus  on  relevance  and 
importance.  Highlight  ‘hidden  gems’  in  postings  to  bring  them  out  of 
obscurity. 

FOCUSING  ON  KEY  POINTS:  Ask  them  to  summarize  or  synthesize.  If 
necessary  eventually  you  can  highlight  key  contributions,  essential  concepts, 
and  connections  so  far.  Bring  to  light  potential  gaps  or  tensions.  Provide  big 
picture,  but  do  not  summarize  in  detail  or  infer  future  directions  - push 
participants  to  draw  these  inferences  and  assessments  by  themselves.  Ask 
them  to  evaluate  the  strength  on  their  ideas,  seek  judgments  and  assessments, 
and  eventually  reach  consensus. 

FULL-SPECTRUM  QUESTIONING:  Use  a wide  questioning  approach 
(Who?  What?  When?  Where?  Why?)  to  push  participants  to  examine  their 
own  personal,  or  collective,  thoughts  and  beliefs. 

■ Push  them  to  go  beyond  by  asking  “so  what?’’  in  a specific  context 

■ Ask  them  to  consider  other  perspectives  that  they  may  not  have 
thought  about  before 

■ Ask  them  to  clarify  or  elaborate  of  their  ideas 

■ Ask  them  explore  their  assumptions  and  sources,  and  provide  a 
rationale  or  examples  for  their  ideas 

■ Ask  questions  to  identify  cause  and  effects/outcomes 

■ Ask  the  team  to  solve  discrepancies  in  the  ideas 

■ Ask  questions  considering  appropriate  action  or  inquiry  especially  if 
the  discussion  is  stuck 

MAKING  CONNECTIONS:  Stretch  the  participant’s  imagination  or 
conceptual  frames  to  consider  obscure  but  essential  similarities.  Moving 
beyond  the  barriers  of  previously  held  beliefs  or  assumptions  that  may  block 
these  connections  across  contexts  or  at  deeper  levels. 

HONORING  MULTIPLE  PERSPECTIVES:  Lay  out  the  landscape  of 
different  views  present  on  the  discussion.  This  is  usually  the  last  stage  before 
the  group  completes  their  final  task. 

* Note:  examples  included  on  the  facilitation  guide  are  not  shown  on  this  table 

Instruments 

All  of  the  instruments  used  in  the  study  were  administered  online.  They  included  a pre -survey,  a post- 
survey, and  two  final  tests. 

Pre-survey.  The  first  section  of  the  pre -survey  consisted  of  two  demographic  items:  name  and  age.  The 
second  section  included  10  five  point  Likert -type  items  where  1 “Strongly  Agree’’  and  5 “Strongly  disagree’’,  that 


SHARPENING  THE  FOCUS 

These  strategies  seek  to  clarify 
ideas  and  find  common 
understanding  among 
participants.  Highlight  relevant 
ideas  and  key  contributions, 
bringing  coherence  and  pushing 
the  dialogue  forward. 


DEEPENING  THE  DIALOGUE 

Build  on  common  ground  by 
promoting  that  participants 
examine  their  own  beliefs  and 
assumptions,  reflecting  on 
perturbations  to  build  new  levels 
of  understanding. 
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were  designed  to  assess  participants'  attitude  towards  the  use  of  DE  (three  items),  their  interest  in  the  topic  (four 
items)  and  their  attitude  toward  current  state  of  K- 12  education  (three  items).  Although  the  internal  consistency  for 
questions  on  the  three  topics  was  not  high  enough  to  validate  the  existence  of  scales,  individual  items  still  provide 
useful  information  about  the  three  indicated  aspects.  The  last  section  consisted  of  6 five  point  Likert -type  items 
designed  to  assess  the  participants’  level  of  trust.  The  trust  items  were  drawn  from  Yamagishi  & Yamagishi’s  (1994) 
scale  of  general  trust,  reaching  in  this  study  an  internal  consistency  of  .92. 

Post-survey.  The  post-survey  consisted  of  31  five-point  Likert -type  items  where  1 “Strongly  Agree”  and  5 
“Strongly  disagree”.  Six  variables  were  measured.  The  first  three  scales  were  designed  to  assess  the  success  of  the 
treatment,  and  measured  participants  perceived  cognitive  facilitation  (nine  items  with  an  internal  consistency  of  .87), 
perceived  facilitator  social  presence  (seven  items  with  an  internal  consistency  of  .89),  and  perceived  facilitator 
motivational  presence  (three  items  with  an  internal  consistency  of  .87).  The  last  three  scales  assessed  overall 
perceptions  of  participants’  experience  during  the  discussion,  and  measured  overall  satisfaction  (six  items  with  an 
internal  consistency  of  .8),  perceived  learning  (three  items  with  an  internal  consistency  of  .79),  and  relevance/use  of 
the  articles  (3  items  with  an  internal  consistency  of  .57). 

Independent  samples  t tests  were  used  to  compare  the  results  of  the  initial  and  final  surveys  reliable  scales. 

Recall  Test.  A recall  test  prompted  the  participants  torememberas  much  as  ideas  they  could  from  the 
articles  they  read  and  from  their  group  discussion.  This  test  was  used  as  a basic  measure  of  learning.  Participants 
were  instructed  not  to  go  back  to  the  articles  or  the  discussion  forum  before  or  during  the  recall  task,  and  they  were 
to  use  at  most  20  minutes  to  complete  the  test  and  to  report  the  time  they  used  to  complete  it. 

Open-ended  Test.  An  open-ended  test  was  used  as  a more  elaborated  measure  of  learning.  It  was  a short  test 
including  four  open-ended  questions  focusing  on  student’s  understanding  of  some  of  the  basic  issues  included  across 
all  K-12  DE  articles  that  were  read. 

Procedure 

The  discussion  took  place  on  the  SiteScape  asynchronous  discussion  forum  (SSF).  Because  the  participants 
had  been  using  the  same  forum  for  their  class  discussions,  they  were  familiar  with  the  interface. 

One  day  before  the  discussion  activity  began  all  participants  were  sent  an  email  message  including  a link  to 
the  online  pre -survey,  and  an  attached  file  including  the  task  instructions.  The  task  instructions  were  also  available 
on  SSF. 

Once  all  the  pre-survey  responses  were  received,  facilitators  posted  their  first  message  on  SSF.  Two 
different  welcome  messages  were  composed  as  the  first  facilitators’  posting,  one  for  the  low  cognitive  facilitation, 
and  one  for  the  high  cognitive  facilitation  condition.  In  addition  to  posting  the  welcome  message  on  SSF,  the  same 
message  was  sent  to  participants  by  email  in  order  to  let  them  know  that  the  discussion  had  started.  After  the 
welcome  message,  each  facilitator  made  up  their  own  messages  according  to  the  flow  of  the  discussion,  and 
according  to  the  experimental  condition  of  the  groups.  Due  to  lack  of  response  from  some  participants,  both 
facilitators  did  not  post  any  message  in  a few  occasions  towards  the  end  of  the  study  in  order  not  to  dominate  the 
discussion.  Facilitators  used  email  only  as  a reminder  to  those  who  were  not  actively  participating.  In  these  emails 
students  were  instructed  not  to  reply  but  to  post  their  contribution  to  SSF. 

Two  days  after  the  discussion  activity  ended,  all  participants  received  an  email  message  including  a link  to 
the  post-survey,  and  another  link  to  the  recall  and  the  open-ended  tests. 

Analysis 

Content  Analysis.  Garrison  et  al.’s  (2001)  content  analysis  categories  were  used  to  assess  the  degree  of 
critical  thinking  exhibited  in  the  online  discussion.  These  categories  divide  critical  thinking  in  four  phases: 
triggering  event , exploration , integration , and  resolution.  Archer  et  al.  (2001)  define  these  phases  as  the  idealized 
logical  sequence  of  the  process  of  critical  inquiry.  The  first  phase,  triggering  event,  is  an  initiation  phase  of  inquiry 
in  which  an  issue  or  problem  is  raised.  In  the  exploration  phase  participants’  brain-storming,  questioning,  and 
exchange  of  information  takes  place.  Moving  beyond  exploration  learners  come  to  the  integration  phase  which  is  an 
iterative  effort  to  construct  shared  meaning  within  the  community  of  inquiry.  The  fourth  phase  is  the  resolution  of 
brought  up  in  the  triggering  event.  The  data  that  did  not  belong  to  any  of  these  categories  were  put  under  a fifth 
category  called  other. 

In  addition  to  critical  thinking  two  other  variables  were  analyzed:  collaborativeness  of  the  participants 
during  discussion  and  the  source  of  support  they  used  as  evidence  for  their  arguments.  Collaborativeness  was 
defined  as  participants’  referring  to  the  arguments  of  their  teammates  in  a capacity  to  extend  their  contributions. 
Collaborativeness  was  coded  into  three  categories  depending  on  the  reference  to  other  participants:  no  reference  to 
others,  referring  to  one  posting  of  a teammate,  and  referring  to  two  or  more  postings  of  teammates.  The  source  of 
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support  was  divided  into  two  categories  according  to  the  types  of  evidences  used  in  the  arguments:  support  by 
referencing  the  readings,  and  support  by  referencing  personal  experience. 

Two  coders  were  responsible  for  the  content  analysis  of  the  discussions.  Sentence-level  idea  units  for 
assessment  of  critical  thinking  and  message-level  idea  units  for  assessment  of  collaborativeness  and  support  were 
used  for  being  the  most  appropriate  in  terms  of  usefulness  and  reliability.  Coders  worked  on  a small  sub-set  of  data 
to  develop  the  rules  to  define  the  analysis  units  and  code  them.  After  there  was  agreement  on  the  definitions  and 
rubrics  of  these  idea  unit  analyses,  the  coders  also  practiced  assigning  data  to  these  categories. 

Since  one  of  the  coders  was  also  a facilitator,  and  in  order  to  avoid  possible  bias,  before  analyzing  the 
discussion  postings  a third  person  removed  the  facilitators’  messages,  changed  participants’  names  in  each  message 
to  a pseudonym,  and  altered  the  order  of  the  messages.  The  two  trained  coders  broke  the  data  into  idea  units 
individually  and  then  compared  their  results.  The  level  of  agreement  for  this  initial  unitization  was  96%  using 
Holsti’s  Coefficient  of  Reliability  measure.  In  all,  the  coders  agreed  on  852  sentence  units  within  the  72  posted 
messages.  After  resolving  discrepancies  between  the  unitization  results  through  discussion,  coders  individually 
coded  a fifty  percent  sample  of  the  messages  into  the  critical  thinking,  collaborativeness,  and  support  categories, 
according  to  the  established  rubrics.  Results  of  coding  were  then  combined  and  contrasted.  Inter-rater  reliability  for 
content  analysis  was  not  as  high  as  expected  and  therefore  average  scores  from  the  two  coders’  results  were  used  for 
analysis  purposes. 

Test  Analysis.  To  assess  participants’  learning,  the  answers  for  the  recall  and  the  open-ended  test  were 
analyzed.  The  results  of  the  recall  test  were  scored  by  two  different  raters.  Using  a set  of  rules  defined  during 
practice  sessions,  raters  compared  the  results  to  a predefined  set  of  142  possible  items  to  recall  that  was  extracted 
from  all  the  articles  used  during  the  discussions.  Raters  reached  a 75.5%  of  agreement  on  their  scoring  results.  The 
results  of  the  open-ended  test  were  also  scored  by  two  different  raters  using  a three  points  rubric  with  two  criteria  for 
each  question,  reaching  a 78.9%  of  agreement.  Independent  samples  t tests  were  used  to  compare  the  results  of  the 
final  tests. 


Results 

Participants’  Incoming  Variables 

Results  on  the  initial  online  survey  (with  a 5 point  Likert  scale  where  l=“Strongly  Agree”  and  5=‘‘Strongly 
Disagree”),  indicate  that  overall  participants’  (n=12)  trust  levels  were  high  (M=1.61,  SD=0.42),  excluding  one 
outlier  (2.5  standard  deviations  from  the  sample  mean)  that  showed  low  trust  levels  (M=3.5)  according  to  the 
instrument.  No  significant  differences  were  found  between  participants  on  the  low  and  high  groups  on  this  variable. 
This  result  is  relevant  considering  the  importance  of  trust  in  student  participation  in  online  discussions  (Yamagishi, 
2001),  and  indicates  that  any  possible  effects  were  independent  of  participants’  trust  levels. 

All  participants  also  showed  a positive  attitude  toward  DE.  When  asked  if  they  agree  with  the  idea  that 
‘‘Distance  education  is  a strategy  that  will  provide  real  educational  opportunities  world  wide,”  eight  chose  the 
‘‘Strongly  Agree”  option  and  four  ‘‘Agree”  (M=1.3,  SD=  0.5).  When  asked  for  their  interest  in  the  discussion  topic, 
again  all  participants  showed  high  levels  of  interest.  To  the  statement,  ‘‘The  use  of  DE  is  a topic  that  interests  me”, 
all  but  one  participant  answered,  ‘‘Strongly  Agree,”  and  the  remaining  answered  ‘‘Agree”  (M=l.l,  SD=0.3).  Finally, 
when  asked  about  their  attitude  toward  K- 12  education,  when  asked  to  react  to  the  following  statement  ‘‘We  are 
doing  all  we  can  to  provide  the  best  education  we  can  in  the  US,”  again  participants  reacted  similarly,  in  this  case 
disagreeing  with  the  statement  (M=4.1,  SD=0.8). 

Overall  these  results  indicate  that  all  participants  were  in  similar  conditions  to  participate  in  the  discussion. 
They  were  all  interested  in  the  topic;  they  valued  the  potential  of  DE  and  had  some  concerns  about  the  current  state 
of  K-12  education.  All  these  are  indicators  that  they  could  have  a live  discussion  on  the  proposed  topic:  the  use  of 
DE  in  K-12  settings. 

Participants’  Perception  of  the  Treatment 

The  three  final  survey  scales  (Table  1)  that  assessed  how  successful  the  treatment  was,  indicate  no 
significant  differences  between  conditions  for  their  perceived  facilitator  social  and  motivational  presence. 
Nonetheless,  despite  of  the  small  sample  size  a significant  difference  t(5)  = 3. 17,  p = .025  was  found  for  the  results 
of  the  perceived  cognitive  facilitation  scale.  These  results  indicate  that  the  intended  manipulation  was  successful  and 
perceived  by  the  participants.  They  should  all  have  the  same  levels  of  social  and  motivational  presence  but  differ  in 
the  level  of  cognitive  facilitation. 
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Table  1:  Treatment  perceptions  for  high  and  low  cognitive  facilitation  conditions  (5  point  Likert  scale  1 =SA  and 
5=SD,  N=7) 


Condition 

Social  Presence 

Motivational  Presence 

Cognitive  Facilitation 

Mean 

1.52 

2.00 

2.15* 

High 

SD 

0.36 

1.20 

0.65 

Mean 

1.75 

2.00 

3.36 

Low 

SD 

0.88 

1.05 

0.37 

* Significant  at  the  .05  level 


Discussions  Analysis 

In  the  context  of  an  exploratory  study,  content  analysis  results  serve  the  main  purpose  of  revealing  patterns 
and  trends  that  should  inform  thinking  for  subsequent  studies.  Thus,  rather  than  reporting  a detailed  statistical 
analysis  we  carried  out  the  content  analysis  focusing  on  conceptual  indicators. 

In  terms  of  critical  thinking,  several  patterns  emerged  indicating  differences  between  treatments.  Table  2 
shows  combined  analysis  by  averaging  the  coders’  ratings  and  the  two  groups  on  each  condition.  Results  indicate  a 
considerably  higher  percentage  of  critical  thinking  by  sentence  units  in  the  high  cognitive  facilitation  groups. 


Table  2:  Percentage  of  idea  units  by  critical  thinking  category.  Combined  coders,  combined  groups 


Content  of  idea 
units 

Low  cognitive  facilitation  groups 
(n=178  units)* 

High  cognitive  facilitation  groups 
(n=248  units) 

Other 

33.99% 

12.40% 

Trigger 

5.90% 

2.72% 

Exploration 

35.96% 

39.72% 

Integration 

17.42% 

42.34% 

Resolution 

6.74% 

2.82% 

* n = mean  number  of  idea  units  at  the  sentence  level  for  combined  treatment  groups 

Regarding  collaborativeness,  results  show  similar  levels  of  collaboration  between  treatments  (Table  3), 
with  a slight  tendency  in  the  high  condition  for  having  in  total  more  references  to  other  participants’  messages. 

Table  3:  Number  of  messages  by  level  of  collaborativeness  - Combined  coders,  combined  groups. 

Number  of  prior  messages  Low  cognitive  facilitation  groups  High  cognitive  facilitation  groups 
referenced  (n=41  units)*  (n=31  units) 

None  62,20%  59.68% 

One  28.05%  33.87% 

Two  or  more  9.76%  6.45% 

* n = mean  number  of  idea  units  at  the  message  level  for  combined  treatment  groups 

Regarding  the  use  of  resources  to  support  discussion  postings  (Table  4),  there  was  an  important  difference 
between  high  and  low  groups,  with  high  cognitive  facilitation  groups  providing  article  based  support  to  their 
postings  more  than  twice  those  on  the  low  cognitive  facilitation  groups. 
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Table  4:  Number  of  messages  by  article  based  support  - Combined  coders,  combined  groups. 


Support  on 

Low  cognitive  facilitation  groups 

High  cognitive  facilitation  groups 

articles  data 

(n=41)* 

(n=31) 

Yes  12.20%  27.42% 

No  87.80%  72.58% 

* n = Mean  number  of  idea  units  at  the  message  level  for  combined  treatment  groups 

Personal  experience  support  was  a coding  decision  made  based  on  whether  messages  showed  evidence  of 
including  examples  from  the  participant’s  experience  with  the  intention  of  using  them  to  support  an  argument.  Table 
5 presents  combined  coder  and  treatment  group  results  for  personal  experience  support,  showing  very  little 
difference  between  the  high  and  low  cognitive  facilitation  groups,  both  exhibiting  personal  experience  support  in 
nearly  30%  of  the  messages. 

Table  5:  Number  of  messages  by  personal  experience  support-  Combined  Coders,  Combined  Treatment  Groups 


Personal  Experience 

Low  cognitive  facilitation  groups 

High  cognitive  facilitation  groups 

Support 

(n=41)* 

(n=31) 

Yes 

30.49% 

29.03% 

No 

69.51% 

70.97% 

* n = mean  number  of  idea  units  at  the  message  level  for  combined  treatment  groups 


Final  Survey  Results 

The  three  final  survey  outcome  variables;  overall  satisfaction,  perceived  learning,  and  self-reported  use  of 
the  articles,  indicate  no  significant  differences  between  conditions.  Nevertheless,  in  all  cases  there  is  a clear 
tendency  for  participants  on  the  high  cognitive  facilitation  groups  to  have  higher  ratings,  as  shown  on  Table  6.  They 
were  more  satisfied  with  the  discussion,  their  perception  of  learning  was  higher,  and  their  self-reported  use  of  the 
articles  was  also  higher. 


Table  6:  Fined  survey  results  by  facilitation  conditions  (5  point  Likert  sccde  1 =SA.  5=SD,  n=7) 


Condition 

Overall  Satisfaction 

Perceived  Learning 

Self  reported  use  of  articles 

High 

Mean 

2.00 

2.00 

1.11 

SD 

0.60 

0.33 

0.19 

Low 

Mean 

2.42 

2.42 

1.67 

SD 

0.52 

0.63 

0.38 

Final  Tests  Results 

In  terms  of  learning  outcomes  (Table  7),  no  significant  differences  were  found,  but  there  is  a tendency  for 
those  in  the  high  cognitive  facilitation  groups  to  have  a higher  number  of  recalled  items  and  a higher  score  on  the 
open  ended  test. 


Table  7:  Learning  outcomes  by  conditions  (n=7) 


Condition 

Number  of  items  recalled 

Test  Score(24  points  scale) 

High 

Mean 

11 

20.38 

SD 

7.35 

2.10 

Low 

Mean 

9.5 

19.13 

SD 

6.03 

1.93 
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Discussion 

The  purpose  of  this  pilot  study  was  to  explore  the  relationship  between  high  cognitive  facilitation,  critical 
thinking,  and  learning  in  online  collaborative  environments. 

According  to  the  results  of  the  initial  survey  all  participants  were  in  similar  conditions  to  participate  in  the 
discussion.  They  were  interested  in  the  topic,  valued  the  potential  of  distance  education  and  had  some  concerns 
about  the  current  state  of  K-12  education.  In  addition,  all  participants  (with  the  exception  of  an  outlier)  showed  high 
trust  levels,  which  was  relevant  for  the  study  considering  the  importance  of  trust  in  student  participation.  All  these 
were  indicators  that  the  manipulation  could  work  and  that  participants  could  have  a live  discussion  on  the  proposed 
topic.  Cognitive  facilitation  was  successfully  manipulated  and  participants  in  the  high  condition  did  perceive  their 
facilitator  as  providing  a significantly  higher  level  of  cognitive  facilitation.  Simultaneously,  there  was  no  difference 
in  the  perception  of  the  facilitator  as  being  friendly  (high  social  presence),  and  providing  good  motivation  for  the 
discussion. 

These  results  are  relevant  for  future  studies  on  cognitive  facilitation,  because  they  show  that  cognitive 
facilitation  can  be  successfully  manipulated.  The  results  also  indicate  that  potential  differences  among  conditions 
were  not  due  to  different  levels  of  personal  interest  or  trust,  neither  to  different  levels  of  social  or  motivational 
presence  on  the  facilitator’s  side. 

The  first  research  question  considered  if  high  cognitive  facilitation  promoted  critical  thinking  among 
participants  or  not.  Based  on  the  results  of  our  analysis,  we  do  see  support  for  a link  between  high  cognitive 
facilitation  and  critical  thinking.  Although  our  sample  size  was  very  small,  we  did  see  evidence  of  more  critical 
thinking  taking  place  among  participants  in  the  groups  receiving  high  cognitive  facilitation  than  among  those 
receiving  low  cognitive  facilitation.  While  the  low  group  had  66%  of  critical  thinking  units,  the  high  group  had  88%. 
Specific  support  to  this  trend  comes  from  the  higher  percentage  of  idea  units  classified  as  integration  for  groups 
receiving  high  cognitive  facilitation.  According  to  Garrisons  et  al.’s  (2003)  model,  integration  is  the  most  important 
category  indicating  critical  thinking  process.  Low  cognitive  facilitation  groups  had  a high  number  of  idea  units 
classified  under  other  (34%),  i.e.  social  or  general  postings  with  no  direct  relation  to  critical  thinking,  and 
concentrated  their  critical  thinking  on  the  two  initial  levels  of  the  process  (trigger  and  exploration  42%). 

Concerning  the  additional  categories  examined,  personal  experience  was  problematic  due  to  low  inter-rater 
reliability.  We  think  there  was  a good  deal  of  confusion  between  the  coders  in  the  definition  used  to  code  this 
category.  This  disagreement  may  also  have  had  an  impact  in  the  critical  thinking  coding  as  well,  since  personal 
experience  was  used  as  specific  indicators  in  Garrison  et  al.’s  examples  (2003).  We  recommend  that  clearer 
definition  and  agreement  be  reached  on  the  meaning  of  personal  experience. 

While  our  inter-rater  reliability  was  lower  than  we  would  have  preferred,  these  indicators  give  us  hope  that 
high  cognitive  facilitation  may  positively  impact  critical  thinking  in  OL,  and  would  therefore  be  a worthwhile 
practice  to  foster. 

There  were  other  relevant  problems  that  arose  in  this  study  that  need  to  be  addressed  and  considered  on 
future  studies.  The  rating  cues  for  critical  thinking,  based  on  the  Garrison  et  al.’s  model  (2003),  were  not  equally 
understood  by  the  two  raters,  and  better  agreement  could  be  reached  by  a clearer  definition  of  the  categories.  In 
particular,  the  ambiguity  between  integration  and  exploration  categories  needs  to  be  clarified.  These  categories 
created  the  most  recognizable  differences  in  coding  between  the  coders. 

One  of  the  coders  in  this  study  was  also  a facilitator  for  two  of  the  groups.  This  coder’s  familiarity  with  the 
discussions,  and  the  potential  of  inferring  context  from  memory,  may  have  had  an  impact  on  the  coding  results, 
particularly  his  higher  count  for  integration.  It  is  recommended  that  future  coding  is  not  done  by  one  of  the  study 
facilitators. 

Context  is  also  an  important  consideration.  We  “sanitized”  the  data  and  changed  the  order  of  the  messages 
so  the  coders  would  not  recognize  the  postings  as  being  from  one  of  the  conditions.  Nevertheless,  the  flow  of  an 
entire  discussion  thread  could  have  provided  clearer  indication  of  the  content  categories  for  individual  messages.  If 
coding  is  done  by  non  facilitators  the  original  flow  of  the  discussion  could  be  used  with  no  foreseen  problem. 

Unitization  was  a struggle  for  this  study.  We  originally  wanted  to  code  at  a “unit  of  meaning”  level,  but  had 
much  difficulty  reaching  agreement  between  coders  on  these  units.  To  reduce  ambiguity,  we  decided  to  code  at  the 
sentence  level.  This  created  much  higher  inter-rater  reliability  in  unitization,  but  introduced  the  problems  of  volume 
of  units  (852  sentences),  and  lack  of  context.  Garrison  et  al.  (2003)  decided  to  unitize  at  the  message  level.  This 
makes  logical  sense  as  most  messages  are  trying  to  accomplish  one  predominant  task.  Additionally,  the  model  we 
used  for  analysis  was  developed  and  previously  applied  for  coding  at  the  message  level,  and  did  not  translate  ideally 
for  applying  it  at  the  sentence  level.  We  would  recommend  future  studies  use  the  message  level  for  unitization,  or 
develop  a better  rating  scale  more  suited  to  the  agreed  upon  units. 

The  second  research  question  considered  if  high  cognitive  facilitation,  and  the  consequent  higher  presence 
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of  critical  thinking  among  participants,  results  in  higher  levels  of  learning  among  participants  or  not.  Results 
indicate  that  there  were  no  statistically  significant  differences  between  the  learning  outcomes  of  the  two  conditions, 
or  between  the  effects  on  perceived  learning  or  satisfaction.  Nevertheless,  results  also  suggest  a consistent  tendency 
in  which  participants  in  the  high  condition  showed  higher  satisfaction,  higher  perceived  learning  and  higher  use  of 
the  articles.  In  terms  of  learning  outcomes,  no  significant  differences  were  found  in  both  the  recall  and  the  open 
ended  tests,  but  again  there  is  a tendency  for  those  in  the  high  cognitive  facilitation  groups  to  have,  in  average,  a 
higher  number  o f recalled  items , and  a higher  score  on  the  open  ended  test. 

The  fact  that  we  had  a small  sample  size  might  have  prevented  us  from  finding  significant  differences 
among  conditions.  Additionally,  the  instruments  might  not  have  measured  learning  in  a way  that  would  help  us 
discriminate  enough  what  participants  learned  during  the  discussion,  and  what  came  from  their  prior  experience. 
Another  factor  that  could  have  impeded  appropriate  discrimination  was  that  each  team  member  in  addition  to  the 
shared  article  received  an  extra  one,  which  made  the  recall  test  scoring  complicated.  Perhaps  an  interesting  topic  in 
which  participants  have  none  or  reduced  previous  experience  could  have  been  selected  to  avoid  this  problem.  Future 
studies  should  consider  these  aspects  when  further  investigating  online  cognitive  facilitation  and  student  critical 
thinking  in  online  environments.  Nonetheless,  as  far  as  we  know  online  cognitive  facilitation  has  not  been 
experimentally  manipulated  before,  and  results  of  this  study  open  an  interesting  avenue  for  future  research  on 
facilitation,  critical  thinking,  and  collaboration  in  online  learning  environments. 
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Abstract 

Although  commonly  considered  something  that  is  for  “after  school’'  and  best  left  out  of  the  K-12 
curriculum,  the  creation  of  comic  books  incorporate  several  skills  from  multiple  academic  curricula.  This  paper 
outlines  two  summer  programs,  one  offered  to  ages  nine  through  15  year  olds  and  another  offered  to  15  through  18 
year  olds.  The  programs  teach  students  how  to  create  their  own  digital  comic  book  and  emphasize  proper  writing 
skills,  storyline  development,  fine  arts  skills,  and  the  use  of  scanning,  graphic  design  software,  and  page-layout 
design. 


Overview 

Although  originating  in  the  very  late  1800s,  comic  books  hit  their  stride  as  a media  in  the  1920s  and  1930s. 
Thus,  reading  comic  books  has  long  been  something  that  school-aged  kids  spend  their  free  time  doing,  but  the  actual 
creation  of  comics  for  these  same  children  hasn’t  been  possible  until  the  availability  of  multimedia  software  and 
hardware  to  the  public  and  to  schools.  Often  thought  of  as  a pastime,  comic  book  development  incorporates  several 
skills  from  multiple  academic  curricula.  Creative  writing,  storyline  development,  various  fine  arts  skills,  graphic 
design,  document  layout,  and  computer  literacy  are  all  involved  with  comic  book  creation.  Students  are  also 
motivated  to  learn  when  it  comes  to  comic  books! 

Papert  (1993)  postulates  that  when  individuals  design  or  create  things  that  are  meaningful  to  them  (or  those 
around  them),  some  of  the  most  powerful  learning  occurs.  In  the  last  decade  or  so,  computers  have  been  developed 
which  have  the  capacity  for  high-level  creation.  Music,  graphics,  video,  simulations,  web  pages,  and  games  are  all 
creations  that  can  be  produced  using  our  current  computer  technology. 

Technology  has  also  integrated  well  with  both  the  fine  arts  and  English  language  arts.  Kaagen  (1998), 
Garner  (1990),  and  Livermore  and  McPherson  (1998)  all  investigate  the  use  of  technology  in  art  in  K-12  schools, 
both  in  the  U.S.A.  and  abroad.  Lawn  (1998)  explores  art  and  technology  in  higher  education  and  finds  the  need  for 
the  use  of  mission  statements  when  combining  the  two  fields.  Sistek-Chandler  (2003),  Groeber  (2003),  and 
McDonnell  (2003)  all  discuss  various  aspects  of  writing  and  technology,  including  strategies  for  writing,  literacy 
software,  and  rubrics. 

Few  have  investigated  or  explored  technology,  fine  arts,  and  English  language  arts.  In  her  work,  Scali 
(1991)  presents  classroom  projects  that  integrate  technology,  fine  art,  aesthetics,  writing,  and  science.  Marshall 
(1998)  describes  how  technology  is  interwoven  through  the  various  fine  arts  curricula  at  Minnesota  State  Arts  High 
School.  Despite  these  examples,  few  have  combined  and  written  about  the  interdisciplinary  combination  of 
technology,  writing,  and  art.  The  current  program  was  an  attempt  to  further  this  important  learning  experience. 

Designed  for  nine  to  eighteen  year  old  male  and  female  students,  the  program  targets  individuals  with  an 
interest  in  comic  books  and  a penchant  for  writing  stories  and/or  drawing.  It  occurs  during  the  summer,  when 
students  have  a break  from  school,  and  encompasses  approximately  40  hours  of  instruction  and  lab  time.  Two 
versions  of  the  program  are  taught  within  the  context  of  two  separate  educational  initiatives.  One  version  of  the 
program  is  offered  during  “Upward  Bound”  to  15  and  18  year  old  students  over  the  course  of  five  weeks.  The  other 
version  of  the  program  is  taught  during  “Summer  Safari”  to  nine  through  15  year  olds  over  one  intensive  week. 

Upward  Bound  is  sponsored  by  the  United  States  Department  of  Education,  within  the  Office  of 
Postsecondary  Education.  It  began  in  1964,  with  monies  from  the  Economic  Opportunities  Act.  Currently,  it  is, 
along  with  Talent  Search  and  Student  Support  Services,  one  of  the  federally  sponsored  educational  opportunity 
initiatives  comprising  the  “TRIO”  Programs.  Upward  Bound  provides  support,  in  the  form  of  workshops,  classes, 
and  programs,  to  individuals  preparing  for  college.  To  be  eligible  to  participate  in  Upward  Bound,  a student  needs 
to  be  from  a low-income  family,  a family  in  which  no  parent  holds  a bachelors  degree,  and/or  a first-generation 
military  veteran.  Literature,  foreign  language,  composition,  laboratory  science,  and  math  instruction  must  be 
provided  in  any  Upward  Bound  program.  Many  other  types  of  courses  or  experiences  are  also  available  (United 
States  Department  of  Education,  2004). 

Summer  Safari  is  a program  sponsored  by  Plattsburgh  State  University  College  Auxiliary  Services  in 
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Plattsburgh,  New  York.  It  is  a not-for-profit  initiative  and  is  entirely  self-supporting  (supplies  and  instructor  fees  are 
supported  by  registration  fees).  Instructors  are  chosen  for  their  expertise  in  specialty  areas  and  a wide  variety  of 
educational  enrichment  courses  and  workshops  are  offered  through  the  program.  As  it’s  name  indicates,  Summer 
Safari  is  only  offered  during  limited  weeks  during  July  and  August  when  students  are  released  from  school  for  an 
extended  break  in  upstate  New  York.  It  is  available  for  five  to  15  year  olds  with  no  economic  or  educational 
prerequisites,  but  specific  courses  may  be  geared  for  a much  more  narrow  age  range. 

The  objectives  of  the  current  program  are  included  below. 

Upon  completion  of  the  program,  students  are  able  to: 

develop  a story  line  for  a comic  book  character  or  group  of  characters; 

write  dialog  for  their  comic  characters,  using  proper  grammar,  sentence  structure,  and  cohesive  ideas; 

sketch  rough  drafts,  or  “thumbnails”  of  their  comic  characters  either  by  hand; 

illustrate  their  stories  through  conventional  methods  of  pencil  and  paper; 

scan  pictures  and  save  them,  in  the  proper  format,  onto  a disk  and  the  hard  drive  of  a computer; 

color  comic  characters  and  scenes  using  a graphic  design  software  such  as  Adobe  Photoshop; 

construct  a portfolio  of  their  work  on  their  comic  book,  documenting  the  various  stages  of  development; 

and 

reflect  on  the  process  of  creating  their  project  either  through  a written  essay  or  verbal  presentation. 

The  highlight  of  this  multidisciplinary  workshop  is  the  seamless,  integration  of  writing,  fine  arts,  and  computer 
technology  to  facilitate  learning  of  curricula  in  each  discipline.  Students  completed  projects  virtually  without  even 
being  aware  of  all  the  content  and  skills  they  were  attaining.  They  were  too  busy  having  fun  to  even  consider  that 
they  were  learning.  As  well,  with  a final-day  presentation  and  display  in  the  local  comic  book  shop,  stakeholders  in 
the  community  are  able  to  instantly  see  the  results  and  value  of  a program  of  this  nature. 

Method 

Participants 

As  per  the  delivery  of  the  workshops,  the  participants  were  grouped  into  two  separate  programs.  The  first 
program.  Summer  Safari,  consisted  of  11  males  and  one  female,  aged  nine-14,  while  the  demographics  of  the  second 
program.  Upward  Bound,  was  comprised  of  three  males  and  seven  females,  aged  15-18  years  old.  Although  both 
groups  utilize  the  same  tools  and  followed  very  similar  instructions.  Upward  Bound  is  geared  towards  introducing 
the  participants  to  career  possibilities,  whereas  Summer  Safari  is  used  to  introduce  and  encourage  the  wonderful 
world  of  comic  books  as  an  art  form. 

Participants  are  expected  to  have  some  prior  knowledge  before  participating  in  these  programs.  They  must  know 
what  a comic  book  is,  show  that  they  have  some  artistic  abilities,  and  have  acceptable  writing  skills. 

Materials 

A group  orientation,  incorporating  DVD  interviews  on  how  professionals  create  comic  books,  was  given  on 
the  first  day.  This  was  then  followed  by  a distribution  of  supplies.  These  supplies  included  specialized  paper  that 
measures  1 l”xl7”  with  a work  area  marked  off  by  “non-repro”  blue  lines;  regular  pencils,  rulers  and  erasers;  regular 
9”xl  1”  drawing  paper;  tracing  paper  and  lined  paper  for  brainstorming,  script  writing  and  [for  Upward  Bound  only] 
journal  entries.  Participants  were  also  given  Crow  Quill  Pen  Holders  and  various  size  Nibs  to  use  with  India  ink  for 
outlining  and  shading  and  Opaque  White  to  make  any  necessary  corrections. 

Paper  portfolios  were  also  distributed  to  the  students  so  they  could  carry  their  artwork  and  supplies  back 
and  forth  to  the  workshops.  Carrying  the  portfolio  also  gave  each  student  a sense  of  “eliteness”.  Portfolios  visibly 
communicate  to  the  general  public  that  the  individual  carrying  a portfolio  as  a “tool-of-the-trade”  is  in  fact  a creative 
and  artistic  individual.  This  acts  as  a confidence  builder  for  many  of  the  participants. 

Heavy  reading  assignments  were  required  for  the  Upward  Bound  participants  to  be  able  to  gain  a better 
understanding  of  the  concepts  behind  the  techniques.  Examples  from  anatomy  books  and  excerpts  from  “how-to” 
books  published  by  DC  Comics,  owned  by  Turner/Time  Warner  Corporation  where  distributed.  For  the  Su  mmer 
Safari,  reading  material  was  lighter,  yet  very  informative.  Publications,  such  as  “Wizard:  The  Comics  Magazine,” 
and  “Sketch”  contain  short  articles  on  drawing  techniques,  inking  pages  layouts,  digital  coloring,  anatomy,  as  well 
as  what  is  needed  to  put  together  a well -organized  portfolio  to  showcase  to  comic  book  companies. 

Since  both  programs  involved  the  use  of  technology,  participants  were  also  given  a list  of  comic  book 
websites  (www.marvel.com  www.dccomics.com  etc)  as  well  as  a list  of  online  resources  (www.bluelinepro.com 
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www.wizarduniverse.com,  etc.).  Several  of  the  resources  provided  have  tutorials  for  many  of  the  techniques 
incorporated  in  this  workshop. 

Procedures 

Each  unit  of  the  program  covered  a sequential  topic  that  is  necessary  for  creating  comic  books.  The 
modules  were  covered  in  a linear  fashion  and  students  were  not  allowed  to  skip  a unit  and  progress  to  the  next 
without  finishing  all  previous  units.  The  units  are  outlined  below: 

Unit  1:  students  flesh  out  their  storylines  and  sketch  initial  drawings  of  their  characters.  They  use  the 
tracing  paper  to  transfer  their  best  images  to  the  specialized  paper. 

Unit  2:  students  outline  their  drawings  with  India  Ink  using  specialized  nibs  and  pens. 

Unit  3:  students  scan  their  comic  book  pages  and  save  them  onto  a floppy  disk/zip  disk/CD  or  a computer 
hard  drive. 

Unit  4:  students  digitally  color,  detail,  and  retouch  their  drawings  using  a graphics  program  such  as  Adobe 
Photoshop. 

Unit  5:  students  finish  digitizing  their  comics  and  begin  compiling  their  portfolios  (both  digital  and  hard 
copy)  for  presentation  the  next  day.  The  newly  created  comic  books  are  presented  to  the  class  as  well  as  any 
parents/guardians/guest  who  wish  to  view  the  creations.  Each  student  takes  home  both  a digital  and  a hard  copy 
version  of  their  comic  book.  For  those  who  give  permission,  their  comic  books  are  also  displayed  in  the  local  comic 
book  shop  for  the  community  to  view  and  appreciate. 

Due  to  the  one-week,  Monday  to  Friday  from  9am-3pm,  schedule  for  the  Summer  Safari  program,  each 
unit  of  this  workshop  took  from  a day  to  a day  and  a half.  Conversely,  Upward  Bound  is  a program  that  runs  for 
five  weeks.  Their  scheduling  consisted  of  one  hour  on  Mondays  and  Tuesdays  and  two  hours  on  Wednesdays  thru 
Fridays.  As  a result,  the  day-to-day  schedule  of  the  Summer  Safari  program  was  then  applied  to  a by-the-week 
schedule  for  Upward  Bound. 

In  both  programs,  participants  learned  what  goes  on  behind  the  scenes  of  creating  a comic  book.  This 
involves  brainstorming  ideas;  writing  stories;  developing  character  descriptions  and  page  layouts;  the  art  of  “inking” 
a page;  and  utilizing  technology  to  scan  the  “inked”  page  and  color  the  page  via  graphic  design  software. 

Criterion  Measures 

A 16-item  Teacher/Course  Evaluation  Survey  assessed  both  sets  of  participant’s  satisfaction  with  various 
aspects  such  as  materials,  content,  instructor,  etc.  of  the  workshop  in  which  they  participated.  The  survey  was  one 
traditionally  used  to  assess  courses  in  the  Upward  Bound  program  and  no  modifications  were  made  to  it.  Permission 
to  use  the  identical  survey  in  the  Summer  Safari  workshop,  for  purposes  of  comparison,  was  obtained  from  the 
Summer  Safari  Director.  The  survey  consisted  entirely  of  selected-response  items,  with  five  response  choices. 
Sample  items  from  the  survey  are  below: 

Please  rate  the  following  items: 

Item  Excellent  Good  Fair  Poor  Bad 

3.  The  instructor  makes  the  subject  interesting. 

15.  The  materials  were  relevant. 

Data  Analysis 

Mean  scores  for  the  Teacher/Course  Evaluation  Survey  were  tabulated  on  a five  - to  one-point  scale 
(Excellent  = 5 points,  Good  = 4 points  Fair  = 3 points.  Poor  = 2 points,  and  Bad  = 1 point).  Means  were  tabulated 
for  both  groups,  Summer  Safari  and  Upward  Bound,  individually  and  collectively. 

Results  and  Implications 

Results  of  the  Teacher/Course  Evaluation  Survey  are  discussed  below  by  mean  scores  for  Upward  Bound, 
Summer  Safari,  and  overall  mean  scores  per  item.  There  were  10  respondents  for  Upward  Bound,  14  for  Summer 
Safari,  and  24  responses  overall. 

As  shown  in  Table  1 below,  overall  both  Upward  Bound  and  Summer  Safari  students  rated  the  instructor 
highly.  Upward  Bound  participants  scored  item  one  “the  instructor  has  thorough  knowledge  of  the  subject,”  item 
nine  “the  instructor  shows  respect  for  students,”  and  item  10  “the  instructor  provides  an  atmosphere  in  which 
students  feel  free  to  ask  questions,”  with  a 5.00.  Summer  Safari  participants  rated  both  item  one  “the  instructor  has 
thorough  knowledge  of  the  subject,”  and  item  13  “the  instructor  has  a sense  of  humor,”  with  a 4.58.  These  scores 
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would  indicate  that,  in  part,  participants  enjoyed  the  courses  due  to  the  individual  that  taught  them.  As  one  would 
expect,  having  knowledge  of  the  subject  matter  (in  this  case  comic  books),  was  found  to  be  important.  However,  an 
instructor  posses  sing  a sense  of  humor  was  also  found  to  be  highly  valuable  by  both  age  and  grade  groups  comprised 
by  Summer  Safari  and  Upward  Bound.  This  finding  is  supported  by  well-established  instruction  and  lesson  design 
models  (Sullivan  & Higgins,  1983).  Some  learning  theories  also  discuss  the  use  of  humor  as  a motivational 
technique  when  teaching.  Clearly,  some  of  the  success  of  the  comic  book  workshops  described  here  has  to  do  with 
the  instructor’s  use  of  humor  with  the  target  age  groups. 

Both  Upward  Bound  and  Summer  Safari  participants  also  scored  item  15,  “the  course  materials  were 
relevant,”  highly  with  scores  of  4.50  and  5.00  respectively.  This  indicates  that  although  these  courses  were  given  in 
the  summer,  participants  found  that  the  assigned  supplementary  readings  were  of  value.  Asking  participants  from 
ages  9-18  to  read  materials  in  preparation  for  a course  may  seem  like  homework  and  is  often  met  with  resistance, 
especially  in  the  summer.  If  participants  feel  that  the  reading  materials  are  important  fora  course  in  which  they 
show  interest,  the  results  here  reveal  that  they  will  find  them  of  value  and  likely  resist  completing  this  task  less. 

Interestingly,  and  perhaps  most  importantly  for  these  participant  groups  is  that  Upward  Bound  students 
rated  item  16  “the  course  information  will  help  me  in  the  coming  year,”  with  a 3.0  while  Summer  Safari  students 
rate  the  same  item  higher  with  a 3.75.  This  finding  is  particularly  intriguing  when  considering  the  age  level  of 
Upward  Bound  is  15-18  years  old  and  the  age  level  of  Summer  Safari  is  nine  to  14  years  old.  Upward  participants 
are  clearly  much  closer  to  entering  and  preparing  for  college  than  are  Summer  Safari  students.  As  well,  overall 
Upward  Bound  students  rated  the  course  more  positively  than  Summer  Safari  students  (4.60  compared  to  4.48). 

This  would  indicate  that  although  Upward  Bound  students  enjoyed  the  course  more  than  did  the  Summer  Safari 
students,  they  felt  it  was  less  applicable  to  their  immediate  future  and  education.  Perhaps  younger  participants  such 
as  the  Summer  Safari  individuals  felt  that  they  had  more  school  and  summer  time  to  explore  creative,  “off-beat,” 
areas  such  as  comic  book  art  and  story -writing  before  getting  down  to  the  “serious”  business  of  college.  If  this  is  the 
case,  it  is  interesting  that  both  younger  and  older  K-12  students  perceive  college  as  an  institution  in  which  creative 
and  unique  interests  cannot  be  explored. 

Table  1 Responses  to  Teacher/Course  Evaluation  Survey 


Item 

Excellent 

Good 

Fair 

Poor 

Bad 

UB 

SS 

UB 

SS 

UB 

SS 

UB 

SS 

UB 

SS 

1.  The  instructor  has 
thorough  knowledge  of 
the  subject. 

5.00 

4.58 

0.0 

.33 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

2.  The  instructor  is  well 
organized. 

5.00 

1.67 

0.0 

2.00 

0.00 

.50 

0.00 

0.00 

0.00 

0.00 

3.  The  instructor  makes 
the  subject  interesting. 

4.50 

3.75 

.40 

.67 

0.00 

.25 

0.00 

0.00 

0.00 

0.00 

4.  The  instructor  keeps 
appointments. 

4.00 

3.33 

.80 

.33 

0.00 

.50 

0.00 

.17 

0.00 

0.00 

5.  The  instructor  is  on 
time  for  class. 

3.00 

2.08 

2.00 

1.67 

0.00 

.25 

0.00 

.17 

0.00 

0.00 

6.  The  instructor  answers 
questions  completely  and 
carefully. 

4.50 

4.17 

.40 

0.00 

0.00 

.25 

0.00 

0.00 

0.00 

.08 

7.  The  instructor  is 
willing  to  give  extra  help 
when  needed. 

5.00 

3.75 

0.00 

1.67 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

8.  The  instructor 
encourages  student 
participation. 

4.50 

2.92 

.40 

.67 

0.00 

.25 

0.00 

0.00 

0.00 

0.00 

9.  The  instructor  shows 
respect  for  students. 

5.00 

3.33 

0.00 

1.00 

0.00 

.25 

0.00 

0.00 

0.00 

0.00 

10.  The  instructor 
provides  an  atmosphere 

5.00 

3.33 

0.00 

.67 

0.00 

.50 

0.00 

0.00 

0.00 

0.00 

in  which  students  feel 
free  to  ask  questions. 
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1 1 . The  instructor  is 
sensitive  to  students’ 
feelings. 

4.50 

2.50 

.40 

2.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

12.  The  instructor 
compliments  students 
who  have  done  well. 

4.50 

2.50 

.40 

1.67 

0.00 

.25 

0.00 

0.00 

0.00 

0.00 

13.  The  instructor  has  a 
sense  of  humor. 

5.00 

4.58 

0.00 

0.00 

0.00 

0.00 

0.00 

.17 

0.00 

0.00 

14.  The  instructor  is  able 
to  relate  well  to  students. 

4.50 

2.50 

.40 

1.33 

0.00 

.50 

0.00 

0.00 

0.00 

0.00 

15.  The  course  materials 
were  relevant. 

4.50 

5.00 

.40 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

16.  The  course 
information  will  help  me 
in  the  coming  year. 

3.50 

3.75 

1.20 

1.33 

0.00 

.25 

0.00 

0.00 

0.00 

0.00 

TOTAL  Mean  Scores 
*N.B.  UB  = Upward  Bound, 

4.60 
SS  = 

4.48  2.48 

Summer  Safari. 

1.28 

0.00 

.31 

0.00 

.04 

0.00 

.01 

Conclusions 

With  growing  initiatives  such  a “No  Child  Left  Behind”  impacting  more  stringent  curricula,  perhaps  PreK- 
12  teachers  are  either  directly  or  indirectly  impressing  upon  their  students  to  “stick  to  the  program”  in  order  to  be 
successful  in  school.  Students  may  interpret  that  there  is  no  room  to  explore  outside  interests  in  school  any  longer 
and  extrapolate  that  impression  through  to  college.  What  a shame  it  would  be  if  the  youth  of  today  no  longer  think 
that  there  is  room  to  explore  creative  interests  in  their  school  curricula. 

Mitchel  Resnick  (2002)  states  that  the  while  the  1980s  included  an  “Industrial  Society”  and  the  1990s 
incorporated  a “Knowledge  Society,”  he  looks  to  the  new  millennium  to  produce  a “Creative  Society.”  He  posits 
that  our  society’s  “success  in  the  future  will  not  be  based  on  how  much  we  know,  but  [rather]  on  our  ability  to  think 
and  act  creatively.”  Clearly,  our  children  are  the  key  to  this  type  of  societal  shift,  especially  as  childhood  is  one  of 
the  most  creative  times  in  our  lives.  Programs  such  as  the  one  described  here,  which  are  multidisciplinary,  integrate 
technology,  and  teach  valuable  academic  and  creative  skills  need  to  be  celebrated  and  treasured. 
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Scaffolding  Pre -Service  Teachers  in  Constructing  a Technology  Enhanced 

Learning  Environment 

Charles  Xiaoxue  Wang 
Georgia  State  University 

Abstract 

One  of  the  critical  tasks  confronting  teachers  today  is  to  integrate  technology  in  their  classroom  to 
facilitate  learning  and  enhance  student  achievement.  This  is  not  an  easy  task.  Teachers  in  these  technology 
integration  courses  face  multiple  challenges  from  pre-sen’ice  teachers:  limited  experience  of  instruction  design  and 
classroom  implementation , limited  skills  of  educational  technology  ( hardware  and  software),  and  limited 
understanding  of  theoretical  framework  behind  technology  integration.  This  paper  shares  with  you  the  experience  of 
scaffolding  pre-sen’ice  teachers  in  constructing  a technology  enhanced  learning  environment. 

Introduction 

Driven  by  increased  use  of  technology  in  K-12  schools  and  NCATE  and  ISTE  standards,  many  teacher 
preparation  programs  offer  courses  on  educational  technology  to  prepare  pre  -service  teachers  in  technology 
integration  (Vannatta  & Beyerbach,  2000).  Teachers  in  these  technology  integration  courses  face  multiple 
challenges  from  pre -service  teachers:  limited  experience  of  instruction  design  and  classroom  implementation, 
limited  skills  of  educational  technology  (hardware  and  software),  and  limited  understanding  of  theoretical 
framework  behind  technology  integration.  This  paper  shares  with  you  the  experience  of  scaffolding  pre -service 
teachers  in  constructing  a technology  enhanced  learning  environment. 

Background  of  technology  integration  cours  e 

Pre -service  teachers  refer  to  those  students  in  various  teacher  preparation  programs  to  obtain  their  teaching 
certificates.  Specifically  in  this  paper,  pre -service  teachers  refer  to  those  students  in  the  Teacher  Education 
Environments  in  Mathematics  and  Science  (TEEMS)  Programs  at  Georgia  State  University.  Now  TEEMS  programs 
develop  to  include  not  only  Mathematics  and  Science  Teacher  Programs  but  also  the  programs  in  English,  RLL  - 
ESOL  (Reading,  Language  and  Literacy  - English  as  a Second  Language),  and  Social  Studies.  After  successfully 
completing  of  the  TEEMS  program,  qualified  candidates  will  receive  initial  certification  to  teach  grades  7-12  in  their 
specific  subject  area. 

According  to  Grabe  and  Grabe  (2004,  xix),  technology  integration  is  to  use  technology  as  a powerful  tool 
“...in  helping  your  students  acquire  the  knowledge  and  skills  of  the  content  area  or  areas  you  will  teach.”  Many 
scholars  (Grabe  & Grabe,  2004;  Jonassen,  et  al.  2003;  Newby,  et.  ah,  1996;  Roblyer  & Edwards,  2000)  advocate  a 
systemic  approach  towards  technology  integration  with  a focus  on  appropriate  and  effective  use  of  technology  in 
facilitating  student’s  learning  and  performance  with  technology. 

IT  7360:  Technology  for  Educators  offered  at  Georgia  State  University  is  a three  credit  hour  required 
course  in  all  TEEMS  programs  that  prepares  pre -service  teachers  to  integrate  technology  in  their  instructions.  It  goes 
beyond  the  basic  use  of  computer  programs.  The  students  in  this  course  are  required  to  complete  a series  of 
assignments  that  involve  using  technology  to  solve  instructional  problems  they  identified.  For  each  assignment, 
students  will  write  documentation  describing  how  the  technology  is  integrated  in  their  instruction  and  reflecting 
upon  their  technology  integration  efforts.  At  the  end  of  the  course,  students  are  required  to  demonstrate  competence 
in  instructional  design  and  technology  integration  through  the  planning  and  creation  of  a technology  enhanced 
learning  environment  in  his/her  subject  which  includes  their  instructional  unit  plan,  lesson  plans,  student-centered 
learning  activities,  and  assessment  plan  (Shoffner,  2004). 

Scaffolding  pre  -service  teachers 

Scaffolding  defined 

Scaffolding  originates  from  the  practice  of  apprenticeship  far  back  in  human  history.  Its  theoretical  roots 
stemmed  from  Vygotsky’s  social  learning  theory  and  “zone  of  proximal  development”.  According  to  Vygotsky 
(1978),  scaffolding  is  defined  generally  as  temporary  support  or  assistance,  provided  by  someone  more  capable  that 
permits  a learner  to  perform  a complex  task  or  process  that  he  or  she  would  be  unable  to  do  alone.  It  is  a special 
assistance  that  helps  learners  to  move  toward  new  levels  of  understanding  of  concepts,  skills,  and  ability  of  solving 
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problems.  Scaffolding  in  this  paper  refers  to  the  assistance  built  in  IT  7360  course  that  helps  pre -service  teachers 
(TEEMS  students)  to  construct  a technology  enhanced  learning  environment  by  integrating  technology  into  their 
classroom  instruction. 

Scaffolding  built  in  IT  7360  includes  three  dimensions  of  technology  integration:  Theoretical  concepts, 
instructional  design  and  development  processes,  and  technology  integration  products.  The  following  reports  what 
the  instructor  did  in  IT  7360  course  to  scaffold  TEEMS  students  in  constructing  a technology  enhanced  learning 
environment. 

Theoretical  concepts  in  technology  integration 

The  theoretical  concepts  refer  to  those  learning  theories,  instructional  design  and  development  models,  and 
the  understanding  of  important  issues  in  and  related  to  technology  integration.  Scaffolding  TEEMS  students’ 
learning  in  this  dimension  involves  exploring,  connecting,  and  ultimately  applying  those  theoretical  concepts  in  their 
technology  integration  efforts.  Some  of  the  major  theoretical  concepts  covered  in  the  course  include  the  following 

• Basic  theories  of  learning  and  implications  for  instruction  design 

• Instructional  design  models  and  principles 

• Learning  environment  and  technology 

• Internet  and  instruction 

• Information  literacy 

• Visual  literacy 

• Technology  integration 

• Copyright  issues  in  technology  integration 

To  explore  these  theoretical  concepts  at  the  beginning  of  the  course  is  to  build  a foundation  for  technology 
integration  efforts.  The  purpose  is  to  seek  the  theoretical  implications  to  guide  TEEMS  students  in  technology 
integration.  Their  actual  applications  in  technology  integration  consolidate  the  theoretical  concepts  explored  and 
make  those  theoretical  concepts  meaningful  to  them  Various  discussions  and  reflections  on  their  technology 
integration  efforts  also  help  connect  those  theoretical  concepts  to  their  personal  experiences  of  technology 
integration.  TEEMS  students’  learning  in  this  particular  course,  starts  with  exploration  of  theoretical  concepts  and 
ends  with  reflections  upon  their  practice  in  technology  integration  as  shown  in  Figure  1. 

Figure  1 : Scaffolding  in  exploring,  connecting,  and  applying  theoretical  concepts 
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Learning  environment  is  one  of  the  theoretical  concepts  that  TEEMS  students  have  difficulty  to  understand. 
Learning  environment,  according  to  Grabinger  (1996),  is  a “broadly  and  carelessly  used”  term  in  educational 
literature.  The  explorations  of  TEEMS  students  both  online  and  from  reading  produced  a long  list  of  learning 
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environments:  “Constructivist  learning  environment  (Jonassen  & Rohrer-Murphy,  1999)”,  “student-center  learning 
environment  (Land  & Hannafin,  2000)”,  “problem  based  learning  environment  (Barrows,  1996)”,  “project-based 
learning  environment  (Kraft,  2003)”,  “Computer-Supported  Intentional  Learning  (CSILE)  (Scardamalia  & Bereiter, 
1991,  1994)” , “online  learning  environment”,  “multimedia  learning  environment”,  and  “web-based  learning 
environment”  to  name  a few.  The  richness  of  literature  on  “learning  environment”  indicates  that  learning 
environment  is  a crucial  factor  in  learning  and  instruction.  However,  it  confuses  pre -service  teachers  at  the  same 
time. 


Students  ’ explorations  bring  forth  multiple  perspectives  on  learning  environment  and  the  guided 
discussions  both  in  class  and  through  WebCt  Course  site  bulletin  board  provide  students  opportunities  to  negotiate 
within  their  learning  community  what  a learning  environment  is  and  how  it  can  be  created.  As  a result,  a working 
definition  of  learning  environment  from  an  instructional  design  perspective  is  created.  A learning  environment  is 
defined  as  a complex  and  dynamic  system  where  people  apply  certain  strategies  and  use  available  resources  to 
achieve  pre -determined  learning  goals.  This  definition  of  learning  environment  clearly  states  its  core  components 
(learning  goals,  people,  resources,  and  strategies  in  a dynamic  system)  to  offer  TEEMS  students  very  concrete 
building  blocks  for  the  construction  of  their  learning  environments.  Figure  2,  Core  Components  of  a Technology 
Enhanced  Learning  Environment  (TELE),  depicts  the  relationships  among  the  four  core  components  of  a learning 
environment  and  the  characteristics  of  TELE. 

Figure  2:  Core  Components  of  a Technology  Enhanced  Learning  Environment  (TELE) 
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Then,  according  to  this  definition,  a TELE  is  designed  and  developed.  After  TEEMS  students  finished  the 
development  of  their  TELE,  they  will  put  it  in  function  and  test  it  in  their  classrooms.  Following  the  test  of  their 
TELE,  TEEMS  student  are  required  to  reflect  on  their  instructional  design  and  technology  integration  efforts.  The 
reflection  at  levels  of  individuals,  groups,  and  the  whole  class  makes  those  theoretical  concepts  of  technology 
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integration  meaningful  to  the  TEEMS  students.  It  connects  those  theoretical  concepts  to  their  personal  experience  of 
technology  integration  which  does  not  only  strengthen  those  theoretical  concepts  but  enhances  their  understanding 
of  those  theoretical  concepts  to  a new  level. 

Instructional  design  and  development  processes 

The  analysis,  design,  development,  implementation  and  evaluation  (ADDIE)  model  of  instructional  design 
are  introduced  to  TEEMS  students  at  the  beginning  of  the  course.  Following  basic  theoretical  concepts,  they  are 
involved  in  actual  instructional  design  and  technology  integration  processes.  To  facilitate  their  technology 
integration  efforts,  both  process  prompts  and  components  displays  are  used  (Lin,  et.  al.  1999).  Process  prompt 
guides  student’s  attention  to  specific  aspects  of  processes  during  technology  integration  while  component  displays 
assisting  students  instructional  design  efforts  to  ensure  the  quality  of  their  instructions  designed.  For  instance,  in  the 
process  of  analysis  of  learners  and  problems,  following  prompts  are  given  to  the  TEEMS  students  in  addition  to  their 
discussions  and  reflections. 


Description  of  learners 

• General  characteristics 

• Specific  entry  competencies 

• Learning  styles 

• Academic  and  social  information  (Kemp,  Morrison,  & Ross,  1998) 

• Implications  for  instructional  design  from  learner  analysis. 

Description  of  problems 

• Problem  phenomena  observed 

• Causes  of  the  problems 

• Impacts  of  problems  on  student’s  learning 

• Proposed  solutions  and  rationale 


Also  to  assist  TEEMS  students  to  go  through  their  instructional  design,  following  matrix  of  designing 
components  is  used. 

A matrix  of  designing  components 

Learning  goals: 

Academic  standards  to  be  addressed: 

Learning  objectives: 

Learners: 

Time  and  Space: 


Learning/ 

Performance 

Criteria  and  Methods 
for  Evaluation 

Learning 

Strategies 

Instructional 

Strategies 

Resources 

Needed 

1 

2 

3 

4 

5 (It  goes  on  as 
needed) 

Both  process  prompts  and  components  displays  clearly  spell  out  detailed  and  required  efforts  in  technology 
integration  processes  that  a TEEMS  student,  who  are  not  experienced  in  instructional  design,  tends  to  ignore.  These 
prompts  also  help  TEEMS  students  see  how  a systemic  approach  can  be  used  to  solve  problems  and  to  produce  an 
effective  instruction  products  (instructional  materials,  lesson  plans,  etc.). 
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The  scaffolding  TEEMS  student’s  going  through  the  instructional  design  and  technology  integration 
processes  is  not  limited  to  providing  guidance  from  the  instructor.  TEEMS  students  in  IT7360  are  usually  diverse  in 
computer  skills  and  instructional  design  experience.  Carefully  pairing  and  grouping  them  by  the  instructor  in 
constructing  a TELE  are  very  important  and  the  scaffolding  provided  by  more  capable  peers  in  instructional  design 
and  development  process  is  highly  recommended  in  technology  integration  courses.  Actually,  pairing  and  grouping 
students  in  instructional  design  and  development  processes  as  a scaffolding  strategy  is  reported  as  “facilitating 
collaboration”  and  “beneficial”  in  TEEMS  students’  reflection  papers. 

hi -time  feedback  provided  during  instructional  design  and  development  processes  and  moderation  on  their 
peer  collaboration  in  constructing  a technology  enhanced  learning  environment  are  also  part  of  scaffolding  built  in 
IT  7360.  Figure  3 illustrate  the  scaffolding  strategies  built  in  instructional  design  and  development  processes. 


Figure  3:  Scaffolding  in  instructional  design  and  development  process 
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Technology  integration  product 

The  ultimate  goal  of  technology  integration  is  to  facilitate  learning  and  improve  performance.  The 
exploration  of  theoretical  concepts  and  the  guided  efforts  in  technology  integration  processes  are  meant  to  enhance 
the  quality  of  technology  integration.  The  systemic  evaluation  of  the  instructional  products  helps  TEEMS  students 
come  to  understand  that  technology  integration  is  not  a simple  use  of  technology  but  a systemic  approach  to 
facilitate  learning  and  performance. 

Scaffolding  in  technology  integration  product  of  IT  7360,  a TELE  constructed  by  TEEMS  students  focuses 
on  appropriate  and  effective  use  of  technology  to  address  instructional  problems  identified  by  TEEMS  students  . Lin 
and  her  associates  (Lin,  et  al.,  1999)  listed  three  important  implications  of  social  learning  theory  for  understanding 
the  reflective  thinking  and  designing  technology-based  environments  that  support  reflection.  First,  reflective 
thinking  involves  social  interactions  because  one  needs  multiple  perspectives  and  feedback  on  their  own 
performance  and  understanding.  Second,  reflective  thinking  is  an  active,  intentional,  and  purposeful  process  of 
exploration,  discovery,  and  learning.  Third,  reflective  thinking  ultimately  involves  understanding  one’s  own  process 
of  learning. 

Scaffolding  in  technology  integration  product  of  IT  7360,  a detailed  instruction  and  specification  on 
technology  integration  product  and  how  it  will  be  eventually  evaluated  is  given  and  discussed  in  class.  Students  are 
advised  to  use  the  evaluation  rubrics  as  a checklist  to  conduct  self  evaluation  of  TELE  they  created  for  their 
students.  Another  scaffolding  strategy  to  enhance  the  quality  of  technology  integration  product  is  to  orient  TEEMS 
students’  technology  integration  effort  around  Quality  Core  Curriculum  (QCC)  by  Georgia  Department  of 
Education  and  other  academic  standards  such  as  Georgia  Technology  Standards  for  Students,  Georgia  Technology 
Standards  for  Teachers,  National  Education  Technology  Standards  for  Students,  and  National  Education 
Technology  Standards  for  Teachers.  This  does  not  only  prepares  TEEMS  students  in  technology  integration  by 
connecting  their  learning  experiences  with  the  actual  tasks  in  the  real  world  but  also  pave  their  way  of  technology 
integration  in  their  future  career. 

Both  in  formative  and  summative  evaluations,  instructional  consistency  and  congruency  are  emphasized. 
TEEMS  students  are  required  to  check  consistency  of  their  instructions  to  ensure  that  specific  entry  competencies 
required  to  perform  learning  tasks  are  built  up.  Additionally,  TEEMS  students  are  required  to  check  if  their 
instructional  objectives,  instruction,  and  assessment  are  congruent.  That  is,  they  are  aligned  so  that  instructional 
objectives,  instruction,  and  assessment  deal  with  the  same  type  of  knowledge  and  skills  and  at  the  same  level  of 
learning  as  revealed  below  by  Figure  4. 
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Figure  4:  Concept  of  instructional  consistency  and  congruency 
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Sharing  technology  integration  products  developed  by  TEEMS  students  in  class  is  another  scaffolding 
strategy.  This  experience  sharing  and  TELE  sharing  in  class  present  multiple  views  and  perspectives  about 
technology  integration  and  how  TELE  can  be  effectively  designed  and  developed.  It  broadens  the  views  of  TEEMS 
students,  creates  a wonderful  opportunity  for  them  to  learn  from  each  other,  enriches  the  contents  for  their 
reflections  and  offers  suggestions  for  improvement  in  their  TELE  construction.  It  is  believed  that  the  appropriate  and 
effective  use  of  technology  for  instruction  is  situational.  Learners,  learning  goals  and  objectives,  learning  resources, 
learning  context,  and  learning  strategies  should  be  considered  as  a whole  learning  system  to  determine  the 
appropriateness  and  effectiveness  of  technology  integrated  in  instruction.  Figure  5 summarizes  the  scaffolding 
strategies  used  to  enhance  the  quality  of  TELE  created  by  TEEMS  students. 
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Figure  5:  Scaffolding  in  technology  integration  products 


Conclusion 

To  prepare  pre -service  teachers  such  as  TEEMS  students  to  appropriately  and  effectively  integrate 
technology  into  their  instructions  is  a very  challenging  task.  The  course  survey,  the  technology  integration  samples, 
documentations,  reflection  papers  yield  many  positive  comments  on  this  three  dimensional  approach  of  scaffolding 
in  this  technology  integration  course  with  TEEMS  students.  The  quality  of  technology  integration  products  of 
TEEMS  students  improves  greatly  when  TEEMS  students  experience  technology  integration  in  these  three 
dimensions. 

Following  are  a list  of  major  activities  that  TEEMS  student  reported  very  beneficial  to  them  in  this  course 
according  to  the  survey  and  reflection  papers . 

• Guided  exploration  of  information  literacy 

o A series  of  group  activities  that  require  students  to  search  for  information,  analyze  and  evaluate 
the  information,  and  synthesize  information  for  class  presentations  on  different  aspects  of 
information  literacy  within  one  hour. 

• Guided  exploration  of  visual  literacy 
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o A series  of  group  activities  that  require  students  to  search  for  information,  analyze  and  evaluate 
the  information,  and  synthesize  information  for  class  presentations  on  different  aspects  of  visual 
literacy  within  one  hour. 

• Defining  learning,  instruction,  and  technology  integration 

o A series  of  both  small  group  and  a class  activities  that  require  students  to  search  for  information, 
analyze  and  evaluate  the  information,  and  synthesize  information  for  class  presentations  on 
learning,  instruction,  and  technology  integration  within  two  two-hour  sessions. 

• Sharing  of  Assignments 

o A number  of  assignments  done  by  TEEMS  students  are  shared  and  discussed  in  the  class. 

• Building  class  knowledge  base 

o Collaboration  among  a whole  class.  It  collects  a various  useful  resources  located  online,  well-done 
class  assignments,  students’  discoveries  and  explorations  of  various  topics  covered  in  the  course, 
etc. 

• Guided  exploration  of  copyright  issues 

o A series  of  group  activities  that  require  students  to  search  for  information,  analyze  and  evaluate 
the  information,  and  synthesize  information  for  class  presentations  on  different  aspects  of 
copyright  issues  within  one  hour. 

Although  positive  effects  of  learning  were  reported  in  the  course  survey  and  student  reflections,  the 
systemic  study  of  this  approach  is  needed  to  provide  empirical  evidence  of  how  scaffolding  built  in  these  three 
dimensions  would  effectively  prepare  TEEMS  students  in  technology  integration  not  only  during  the  semester  when 
they  are  taking  the  course  but  also  later  in  their  actual  teaching  career!  Furthermore,  scaffolding  in  these  three 
dimensions  is  one  way  to  meet  challenges  teacher  educators  face  in  technology  integration.  Through  the  paper,  I 
sincerely  hope  to  exchange  ideas  with  other  teacher  educators  and  seek  more  effective  ways  to  prepare  pre -service 
teachers  in  technology  integration. 
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Abstract 

Online  learning  has  become  a new  paradigm  in  education,  but  very  few  research-based  studies  have 
addressed  the  multicultural  aspects  of  online  learning.  Using  interviews  and  obsen’ations,  this  study  examines  the 
perceptions  of  Asian  students  while  taking  synchronous  online  courses  in  the  United  States.  The  specific  questions 
explored  in  this  study  are:  (1)  How  do  Asian  students  shift  their  learning  styles  from  physical  classrooms  to 
synchronous  online  learning  environments?  (2)  What  are  Asian  students'  opinions  of  synchronous  online  courses? 
(3)  How  do  Asian  students  interact  in  synchronous  online  learning  environments?  (4)  How  do  the  instructional 
dimensions  of  synchronous  online  courses  accommodate  the  needs  of  Asian  students? 


Introduction 

Many  institutions  of  higher  education  in  the  United  States  have  entered  the  distance  learning  arena  because 
it  makes  economic  sense,  and  also  provides  more  educational  opportunities.  An  important  premise  under  this  new 
instructional  approach  is  that  these  educational  opportunities  should  be  responsive  to  the  needs  of  students  and  the 
world  in  which  they  work  and  live  (Palloff  & Pratt,  1999).  As  a result,  the  instructional  design  of  online 
environments  should  be  informed  by  the  needs  and  special  characteristics  of  the  learners.  International  students  are 
one  of  the  groups  that  need  special  attention  because  they  are  from  different  cultures.  How  to  accommodate  cultural 
considerations  in  online  learning  environments  has  become  an  increasingly  important  issue  to  explore,  especially  as 
online  courses  extend  their  markets  around  the  globe.  This  study  focuses  on  international  students  from  Asia  already 
studying  in  the  USA  with  the  goal  of  analyzing  their  perspectives  of  taking  synchronous  online  courses  in  the  United 
States.  The  synchronous  (real  time)  online  course  uses  a web-based  tool  which  includes  both  audio  and  textual 
communications.  These  students  are  in  the  ironic  position  of  traveling  halfway  around  the  world  for  higher 
education  and  then  finding  themselves  enrolled  in  synchronous  online  courses.  The  findings  of  the  research  are 
expected  to  contribute  to  the  literature  about  multicultural  considerations  in  synchronous  online  learning 
environments  as  well  as  to  provide  suggestions  for  enhancing  the  design  and  implementation  of  online  courses  in 
general. 


Toward  multicultural  considerations 

The  computer  and  networked  technologies  have  opened  the  way  for  educators  to  reach  the  learners  “beyond 
brick  and  mortar”  campuses  (Mclsaac,  2002).  Potential  learners  are  not  just  from  local  areas,  they  could  be  from 
other  countries  around  the  world.  Harasim  (2000)  has  proclaimed  that  online  learning  is  no  longer  peripheral  or 
supplementary;  it  has  become  an  integral  part  of  mainstream  society.  A new  paradigm  in  learning  is  just  about  to 
emerge. 

Following  this  trend,  research  in  distance  education  has  reflected  the  rapid  growth  in  the  field.  However, 
the  literature  has  indicated  little  published  research  on  the  cultural  aspects  of  online  learning  and  teaching,  and  there 
have  been  few  research-based  studies  (Gunawardena,  Wilson,  & Nolla,  2003).  Matsumoto  (1996)  described  culture 
as  “the  set  of  attitudes,  values,  beliefs,  and  behaviors  shared  by  a group  of  people,  but  different  for  each  individual, 
communicated  from  one  generation  to  the  next”  (p.  16).  It  is  hard  to  notice  how  culture  influences  people  without 
seeing  actual  artifacts.  Artifacts  such  as  foods  and  advertisements  are  shaped  by  culture,  but  this  process  is  usually 
activated  unconsciously  when  people  think  and  interact  with  the  external  world.  Henderson  (1996)  has  reminded  us 
that  instructional  design  is  a product  of  culture,  and  it  cannot  and  does  not  exist  outside  of  a consideration  of  culture. 
McLoughlin  ( 1999)  also  stated  the  need  to  ensure  the  coverage  of  every  culture  is  important  in  the  beginning  of 
adopting  an  instructional  design  model.  While  accommodating  cultural  differences  sounds  good  in  theory,  a 
dilemma  happens  when  there  is  a conflict  between  the  cultures  under  consideration.  These  are  the  challenges 
mentioned  by  Reeves  and  Reeves  (1997)  when  the  core  pedagogical  values  in  one  culture  are  culturally 
inappropriate  in  another.  This  dilemma  not  only  causes  uncomfortable  feelings  among  learners,  but  sometimes  it 
will  cause  serious  misunderstandings.  McLoughlin  (1999)  warned  that  even  seemingly  innocent  design  features  in 
the  learning  environment  may  nevertheless  cause  cultural  misunderstanding.  For  example,  the  color  of  text  in  an 


832 


online  learning  environment  is  critical  to  Chinese  students  (Tu,  2001).  Chinese  interpret  red  text  as  a warning,  but 
the  instructor  or  instructional  designer  may  use  red  just  to  emphasize  the  content. 

Ku  and  Lohr  (2003)  in  their  case  study  regarding  Chinese  students’  attitude  in  online  learning  experience 
suggest  that  teachers  should  attempt  to  increase  students’  self-confidence  and  motivation  in  the  early  stage  of  the 
course,  design  small  group  activities  for  giving  feedback,  maintain  a self-directed  design  of  the  learning 
environment,  provide  varies  opportunities  for  writing  and  reading,  and  if  possible,  encourage  face  to  face 
interactions  among  group  members  and  the  instructor.  Based  on  their  teaching  experience,  some  researchers  have 
also  concluded  that  social  and  cultural  understandings  need  to  be  explicitly  and  honestly  discussed  in  a class  (Chen, 
Mashhadi,  Ang,  & Harkrider,  1999).  In  addition,  the  cultural  factors  in  learning  environments  need  to  be  negotiated 
carefully  because  students  from  different  cultural  background  usually  have  different  relationships  among  teachers 
and  students  as  well  as  different  ways  to  comment  on  the  work  of  others  (Schallert,  Reed,  & D-Team,  2003). 
International  students  are  often  unwilling  to  address  the  teacher  or  other  students  in  the  class  because  they  are 
unfamiliar  with  the  ways  of  holding  public  discussions  in  the  class.  Schallert  et  al.  (2003)  also  mentioned  that 
teachers  need  to  encourage  international  students  to  keep  writing  in  an  online  class  and  to  lower  their  internal  editor 
when  composing  their  messages.  The  instructors  need  to  provide  some  guides  for  students  to  appropriately 
communicate  online  such  as  how  to  use  common  paralanguage  (Tu,  2001). 

Additionally,  these  online  learning  experiences  are  also  influenced  by  the  perceptions  they  hold.  Research 
has  shown  that  not  only  the  online  environment  itself,  but  also  the  perceptions  of  students  will  influence  their 
interaction  in  an  online  learning  environment  (Tu,  2001).  Different  social  contexts  nurture  various  perceptions  that 
hold  in  students’  mind.  Tu  also  provided  an  observation  that  Chinese  students  maintain  tradition  reactions  during 
online  learning  interactions.  For  example,  “saving  face’’  remains  just  as  important  in  an  online  course  environment 
as  it  does  in  a physical  classroom.  In  an  ongoing  longitudinal  study.  Pan,  Tsai,  Tsai,  Tao,  and  Cornell  (2003)  have 
tried  to  analyze  the  elements  embedded  in  Confucian  pedagogy  and  Western  pedagogy,  and  then  seek  if  there  is 
symbiosis  or  asymbiosis  for  these  different  pedagogies.  From  their  draft  report,  some  preliminary  results  can  be 
applied  to  online  environment  course  design.  For  example,  one  principle  they  espouse  is  “encourage  Asian  students 
to  not  work  together  in  the  same  project  groups,  thus  provide  the  Asian  students  with  direct  opportunities  and 
challenges  in  using  English.’’  (p.  324). 

Other  than  cultural  considerations,  Asian  students  also  face  language  barriers  in  their  learning.  However, 
research  (Schallert  et  al.,  2003)  found  that  despite  the  constraints  on  international  students’  input  in  both  oral  and 
written  discussion,  computer-mediated  discussion  offered  increased  opportunities  for  international  students  to 
contribute  ideas  in  the  class.  To  what  extent  Asian  students  improve  their  learning  experience  in  online  classes  is 
unknown,  but  it  is  clear  that  the  online  learning  environment  does  have  impact  on  their  learning.  In  addition,  most  of 
the  studies  which  have  been  done  so  far  were  focused  on  asynchronous  online  learning  environments.  What  would 
be  the  differences  if  the  online  environments  are  synchronous  rather  than  asynchronous  for  Asian  students?  Do  the 
advantages  or  disadvantages  of  Asian  students  in  online  learning  environments  remain  the  same  in  both  synchronous 
and  asynchronous?  More  studies  are  needed  to  address  this  issue. 

Research  Questions 

In  order  to  better  understand  cultural  factors  in  synchronous  online  learning  environments,  this  study  adopts 
qualitative  methods  to  examine  how  the  synchronous  online  learning  environment  and  individual  cultural 
differences  interact  together,  and  more  specifically  how  Asian  students  adapt  from  learning  in  traditional  face-to- 
face  classrooms  to  learning  in  synchronous  online  learning  environments.  The  synchronous  online  learning 
environment  in  this  study  is  a two-way  (audio  and  instant  message)  communication  web-based  environment. 

Students  can  hear  the  instructor’s  lecture,  speak  to  the  group,  type  instant  messages  and  view  the  PowerPoint  slides 
through  the  web-based  tool.  Except  for  the  first  class,  the  course  is  conducting  completely  online  synchronously. 
Four  specific  questions  are  explored  in  this  study: 

■ How  do  Asian  students  shift  their  learning  styles  from  physical  classrooms  to  synchronous  online  learning 
environments? 

■ What  are  Asian  students'  opinions  of  synchronous  online  courses? 

■ How  do  Asian  students  interact  in  synchronous  online  learning  environments? 

■ How  do  the  instructional  dimensions  of  synchronous  online  courses  accommodate  the  needs  of  Asian 
students? 
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Methodology 


Participants 

Eight  participants  are  expected  in  this  study.  However,  because  the  course  is  only  offered  in  the  spring 
semester,  1 have  only  been  able  to  find  three  participants  to  join  in  this  study  so  far.  All  of  these  participants 
participated  in  a face  to  face  interview,  and  one  of  them  also  took  part  in  the  participant  observation.  The 
participants  are  international  students  who  are  from  Korea,  Singapore,  and  Taiwan,  and  they  are  or  have  been 
enrolled  in  the  graduate-level  synchronous  online  course  on  which  this  study  is  focused.  Data  collection  and  analysis 
is  continuing  at  this  time.  These  participants  so  far  are  all  Masters  students  and  they  are  required  to  take  this  course 
in  order  to  complete  their  Masters  degree.  The  time  they  have  been  in  the  United  States  ranges  from  less  than  one 
year  to  almost  two  years.  Additionally,  for  one  of  them,  this  is  the  first  time  taking  an  online  course  in  the  United 
States.  Other  demographic  factors  like  gender  or  age  are  not  considered  in  this  study.  However,  this  study  will 
attempt  to  have  equal  number  of  the  partic  ipants  in  both  genders  and  nationality. 

Data  collection 

This  study  uses  semi -structured  interviews  and  participant  observations  to  generate  data.  Three  different 
audio  recoded  semi -structured  interviews  were  conducted  during  March  and  April,  2004,  and  another  four  will  be 
conducted  during  the  spring  semester,  2005.  The  interviews  are  face-to-face  and  followed  the  interview  protocol. 
Each  interview  is  around  one  hour  long.  In  addition,  a one  hour  long  participant  observation  was  conducted  during 
the  spring  semester,  2004,  and  another  two  are  ejected  to  be  conducted  during  spring  semester  2005.  In  the 
observation,  I was  virtually  and  physically  staying  with  the  participants  to  observe  their  interactions  both  in  the 
online  classroom  and  outside  the  online  classroom  during  a one-hour  period  in  the  class.  I recorded  the  observations 
in  the  field  notes,  and  a short  informal  interview  is  being  conducted  to  clarify  the  findings  in  the  observations. 

Data  analysis 

This  study  adopts  part  of  Carspecken’s  (1996) reconstructive  method  to  analyze  the  interview  transcriptions 
and  observation  field  notes.  The  first  step  is  to  employ  low-level  coding  to  make  out  both  frequently  mentioned 
concepts  and  unusual  ones.  In  this  study,  we  use  one  of  the  interviews  to  generate  the  primary  codes.  The  process  is 
to  go  through  the  whole  transcription  and  list  examples  which  can  support  these  primary  codes.  Then,  using  these 
emerging  codes  as  the  pre -determined  codes  in  other  interviews,  those  interviews  are  analyzed  to  find  more 
examples.  Meanwhile,  if  there  are  any  special  events  or  frequently  mentioned  concepts  in  other  interviews  that  did 
not  appear  already;  new  codes  will  be  added  with  the  examples.  In  addition,  some  statements  related  to  self 
identification,  such  as  “I  am  a person  who  likes  to  try  different  experiences”  are  also  listed  in  the  document. 

After  the  primary  codes  with  supported  examples  are  marked  out  from  these  transcriptions,  the  internal 
consistency  check  is  performed.  This  process  is  to  make  sure  that  the  examples  really  support  the  codes,  and  to 
determine  whether  any  new  codes  or  sub-codes  should  be  distinguished  from  the  original  primary  codes.  For  the 
participant  observation,  similar  processes  are  employed  into  the  data  analysis.  Specifically,  the  goal  is  to  make  out 
the  patterns  of  online  interaction  and  strategies  the  participant  used  during  the  online  course.  The  results  found  in  the 
observations  are  incorporated  into  the  findings  in  the  interviews. 

Using  this  coding  scheme,  the  results  of  the  interviews  and  observations  are  being  used  to  reveal  the  themes 
and  patterns  in  the  participants’  perspectives  of  their  experiences  in  the  synchronous  online  learning  environment. 

Findings 

Several  primary  codes  and  categories  are  emerging  from  the  data.  By  integrating  these  categories  and 
comparing  the  codes  constantly,  the  data  are  synthesized  into  several  main  themes.  The  findings  below  are 
organized  by  these  themes  and  interview  excerpts  are  provided  to  support  the  findings. 

Opinions  of  synchronous  online  courses 

Synchronous  online  learning  is  useful  because  the  participants  do  learn  things  from  it  and  the  environment 
is  actually  better  than  they  originally  expected.  Originally,  the  participants  thought  that  synchronous  online  courses 
might  be  just  require  listening  to  lecture  and  exchanging  ideas  in  a simple  way,  like  one  participant  says: 

“ Originally  I think  that  maybe  just  describe  something  and  o.k.  I can  get  some  information  and  hear 
some  some  listen  to  somebody 's  thinking,  just  that.” 

However,  after  experiencing  this  synchronous  online  course,  they  found  it  is  better  than  their  original  expectation. 

As  one  participant  says : 
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“ And  this  environment  I think  is  .um..  Also  better  than  I expected.  The  sound  quality  and  the 
message,  I think  that’s  good  enough  for  learning  than  I expected.” 

Although  the  participants  think  synchronous  online  learning  is  a useful  way  of  learning,  they  still  think  students 
need  to  meet  face  to  face.  One  participant,  after  mentioned  several  benefits  of  synchronous  online  courses  still 
pointed  out  the  concern  that  “ Basically , the  students  need  to  come  face  to  face.”  More  critically,  one  participant 
addressed  that  synchronous  online  course  cannot  substitute  traditional  course.  As  he  says: 

“7  think  it  a good  way  of  thinking,  but  it  can... I don ’t  think  it  can  substitute  traditional  course.” 

He  further  addresses  the  point  that  the  use  of  synchronous  online  technology  should  depend  upon  the  course 
contents.  As  he  says: 

“We  cannot  apply  online  course  to  all  courses.  It  should  be  depended  on  the  property  of  the 
material.” 

In  addition,  another  participant  pointed  out  that  the  online  learning  environment  is  not  absolutely  a learner-centered 
design.  It  will  depend  on  the  instructor’s  pedagogy.  As  he  says, 

“...rather  it  ( synchronous  online  course)  is  teacher-centered  or  student-centered  actually  depends  on  the 
lecturer  himself... I think  it’s  not  so  much  about  the  medium,  It’s  more  on  the  methodology  the  teacher 

adopts.” 

Learning  experiences  in  synchronous  online  courses 

The  participants  feel  the  learning  experience  in  the  synchronous  online  course  is  enjoyable  and  convenient, 
because  they  do  not  need  to  come  to  the  physical  campus  and  can  be  more  relaxed  during  the  class.  As  one 
participant  says: 

“Learning  experience,  in  a way  for  me,  it  is  quite  an  enjoyable  one..  .The  class  is  conducted  online 
and  it’s  pretty  convenient  for  me  ...  ” 

Although  the  participants  feel  more  relaxed  physically,  this  does  not  mean  that  they  pay  less  attention  during  the 
class.  One  participant  addressed  his  point: 

“We  can  be  very  casual,  because  you  are  not  sitting  in  a face  to  face  class.  So,  we  can  be  very  causal, 
and  very  at  easy.  Yeah,  having  our  snack  at  the  same  time.  Ok.  All  right,  that's  one  thing,  you  can  be  veryrelaxed. 
Relaxed  doesn  ’t  mean  not  pay  attentions,  but  at  least  physically  you  can  be  very  relaxed  and  listening” 

Another  thing  which  is  convenient  for  the  participants  is  that  the  participants  can  review  the  previous  course  through 
the  archives  in  the  web  site.  The  participants  can  listen  to  the  course  one  more  time  if  they  need.  As  the  participant 
describes: 

“ But..um..it  is  useful. .um.. it  was  useful  for  me  to  review  the. .review  the  content  on  the  website .” 

Besides,  the  participants  feel  nervous  when  they  need  to  speak  out  their  ideas  online,  because  they  are  afraid  of 
making  language  mistakes  and  other  classmates  might  not  understand  them.  As  they  describe: 

“Speak. ..I. ..will  be  a little  nervous.  Yeah,  because  you  know  many  potential  students,  classmates  are 
listening  to  you.  Yeah,  so  lam  afraid  that  I was  saying  something  wrong  or  umm...  grammar...  grammatically 

incorrect.  ” 

“I  found  sometimes  I still  stumbled  a lot  when  l need  to  present  my  ideas. ..You  will  be  thinking  if  I 
speak  in  a correct  way,  so  that  the  rest  of  the  class  knows  what  I am  talking  about.” 

In  addition  to  the  feeling  of  taking  synchronous  course,  the  participants  think  they  lack  the  interactions  with  other 
students  except  for  the  students  in  their  own  sub-group.  There  are  not  much  inter-group  interactions,  and  as  a result, 
the  whole  class  lacks  the  sense  of  learning  community.  As  they  describe: 

“...actually  we  don ’t  have  many  interactions  with  rest  of  group  members  except  within  our  own  group” 
“There  was  no  chance  to  communicate  with  them  ( other  students)... we  use  to  communicate  within  the 

team.” 

“ Not  enough  to  build  a learning  community.  ” 

Challenges  in  synchronous  online  courses 

A big  challenge  mentioned  across  the  participants  is  the  communication  issue.  The  synchronous  online 
communication  lacks  the  facial  expression,  and  this  makes  it  harder  for  the  participants  to  understand  what  their 
classmates  are  saying.  This  relates  to  the  language  barrier  issue,  but  seeing  people  would  help  them  to  understand 
different  accents  or  guess  what  their  classmates  are  saying.  As  one  of  the  participants  describes: 

“...it’s  always  easier  for  us  to  meet  face  to  face  because  meeting  face  to  face  at  least  you  are  able  to  get 
expression,  and  another  person  you  are  able  to  tell  sometimes  even  though  you  are  not  be  able  to 

understand  the  words,  but  you  are  able  to  tell  how  he  or  he  is  expressing  at  that  moment.” 
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In  terms  of  communication,  synchronous  online  learning  makes  it  difficult  for  students  to  illustrate  their  ideas  or  do 
a live  demonstration.  Especially  when  the  participants  cannot  articulate  their  ideas  clearly  by  speaking,  being  able  to 
draw  a graphic  would  help  to  express  their  thinking.  This  challenge  as  one  participant  says: 

“ Online  is  pretty  difficult  to  so  call  draw  your  idea  for  the  three  of  us,  but  if  we  meet  face  to  face  we  are 
able  to  draw  down  our  ideas  as  you  draw  things  out.  To  draw  a diagram  and  so  on  so  far.  ” 

In  addition,  in  the  synchronous  online  course  asking  questions  is  not  as  easy  as  in  the  traditional  face-to-face  course. 
The  instructor  might  not  see  your  messages,  or  you  have  to  wait  for  a while.  One  participant  describes: 

“...  online  class  when  you  need  to  carry  out  a doubt,  you  have  to  take  turns.  You  have  to  give  cue... and  it’s 
just  a long  cue  at  a time  to  just  ask  a question.  So,  at  least  at  face  to  face  you  have  eye  contacts,  and  easier 
to  get  attentions  of  the  lecturer” 

Besides,  there  are  fewer  opportunities  for  informal  interactions  in  synchronous  online  courses,  and  thus  it  is  harder 
for  classmates  to  become  friends  in  synchronous  online  courses.  One  participant  mentioned  this  in  the  interview: 

“ Because  many  informal  interactions  are  sitting  in  traditional  class...,  so  there  is  much  less  chances  for  us, 
students  to  interact  informally  (in  online  class).” 

Another  challenge  found  both  in  the  interview  and  participant  observation  is  that  the  participants  cannot  listen  to  the 
lecture  and  type  or  read  instant  messages  at  the  same  time.  As  one  participant  describes: 

“ I... will  have  to  concentrate  to  what  Dr.  XXX  said,  so  I listen  to  him.  If  I umm...if  I pay  attention  to  the 
message  below  here,  I cannot  read  and  listen.  You  will  be  confused.” 

Learning  barriers  in  synchronous  online  courses 

Language  is  biggest  problem  for  the  participants.  It  affects  not  only  their  understanding  of  others,  but  also 
their  expression  of  ideas.  For  example,  there  are  communication  problems.  As  one  participant  describes  his 
experience: 

“ There  are  times  when  I need  to  ask  certain  questions  I found  that  there  are  times  when  my  lecturer 

might  not  surly  understand  or  misunderstand  what  I am  speaking.  So  there  are  certain  times  I need  to  rephrase 
few  times.  ” 

Other  common  issue  mentioned  by  the  participants  is  that  other  students  sometimes  speak  too  fast  for  them  to  catch 
what  they  said.  In  addition,  as  noted  above  in  the  paper,  sometime  the  participants  cannot  understand  what  other 
students  say  because  of  their  accents.  This  problem  was  mentioned  by  a participant: 

“ Some  students  they  speak  so  fast  that  I cannot  fully  understand,  so  I have  to  guess.  1 don ’t  like  the 
feeling  to  guess  everything.  ” 

In  addition,  due  to  the  language  barrier,  the  participants  tend  to  be  more  reserved  in  the  class,  and  the  consequence  is 
that  they  seem  to  be  not  really  participated  in  the  class.  As  one  participant  describes: 

“7  think  yes  the  biggest  constraint  will  be  the  language,  because  if  you  are  not  at  easy  with  that 
language  that  being  used  so  what  happen  was  that  you  tend  to  be  more  reserved,  and  then  you 

cannot  do  discuss  more.  So,  what  happened  is  that  then  you  will  seem  to  be  as  not  really  participated  in  that  class.  ” 

Ways  to  improve  synchronous  online  learning  experience 

The  participants  would  like  to  establish  some  regular  face-to-face  meetings  during  the  course,  so  that  they 
can  know  each  other  better  and  clarify  some  communication  problems  during  those  meetings.  As  one  participant 
describes: 

“ At  the  midpoint,  the  lecturer  might  consider  to  gather  all  of  us  to  meet  face  to  face.  Maybe  just  in  the 
midpoint  we  probably  as  a group  hang  out.  Those  strategies  we  can  communicate  better.  Just  to  take  feedback 
of  what  difficulties  everyone  has  in  communicating  in  online,  so  that  we  can  all  come  out  with  something. 

In  addition,  some  asynchronous  techniques  could  be  adopted  so  that  topics  can  be  followed  up  and  the  participants 
would  have  more  time  to  write  and  think  about  their  responses.  For  example,  adopting  an  online  forum  might  be  a 
good  way  to  do  that.  As  one  participant  says: 

“/  always  have  the  idea  that  a forum  can  actually  help.  Do  encourage  participants  in  the  online  class  to 
post  their  ideas,  because  in  a forum.  Questions  and  doubts  can  always  be  followed  up.  Everyone  can 
come  in  holds  a view  on  a certain  idea,  and  at  anytime.  ” 

With  respect  to  what  the  instructor  can  do,  the  participants  think  maybe  there  should  be  more  inter-group  activities 
so  that  they  can  experience  more  interactions  with  other  classmates.  As  one  participant  describes: 

“/  think  the  teacher  could  have  the  groups  in  the  class  interact  more  with  each  other,  because  I think 
although  I have  difficulty,  1 don  't  know  much  about  other  groups.  We  are  basically  very  much  in  our  group.  ” 
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Other  potential  way  to  improve  the  participants’  learning  in  synchronous  online  learning  is  to  incorporate  more 
multimedia  elements,  such  as  movies.  The  participants  also  mention  the  possibility  to  have  a video  conference  in  the 
class.  As  the  participant  describes: 

“If  it  can  incorporate  more  multimedia,  .um...  materials,  such  as  movies  or  such  as  what  I just  scud. 
Like  video  conference.  It  will  provide  more  information  other  than  just  the  white  board.” 

Cultural  factors  and  other  considerations  in  taking  synchronous  online  courses 

Very  few  findings  have  been  explicitly  identified  as  cultural  factors  in  this  study.  One  thing  mentioned  by 
one  participant  is  that  he  seems  to  be  more  reserved  compared  to  his  American  classmates.  As  he  describes: 

“7  only  see  it 's  only  appropriate  for  me  to  ask  question  when  it ’s  very  quiet,  No  one  else  is  asking  question 
...  I think  coming  from  Asian  background,  somehow,  just  more  reserved.  In  that  sense  very  different  from  USA 
counterparts.” 

Another  participant  also  mentioned  that  it  is  nearly  impossible  for  him  to  understand  the  conversation  when 
the  content  is  related  to  American  culture,  even  if  he  knows  every  word  being  used  in  the  conversation.  He  provides 
an  example: 

“ Some  American  idiom  stories  and  I totally  have  no  ideas  what  he  is  talking  about.  Even  I 
understand  the  terms  being  used,  I still  don ’t  understand  the  whole  story.  Additionally,  when  they  say 
some  jokes  I cannot  understand  neither,  even  I know  what  the  words  mean.” 

Since  the  language  barrier  is  obviously  a big  concern  while  taking  a synchronous  online  course,  I also 
asked  a hypothetical  question  to  find  out  what  factors  would  influence  their  decision  making  about  whether  to  take  a 
synchronous  online  course  if  English  comprehension  was  not  a problem  for  them.  The  results  show  that  the 
participants  think  this  will  depend  on  the  location,  cost,  course  contents,  and  their  personal  schedule.  If  they  lived  far 
away  from  the  campus,  they  might  consider  taking  online  course.  If  the  online  course  were  cheaper  than  face  to  face 
course,  they  might  consider  taking  the  online  course.  If  the  course  contents  were  difficult  and  need  more  thinking 
such  as  statistic  course,  they  might  consider  taking  it  face  to  face.  In  addition,  the  decision  will  depend  on  their 
personal  schedule.  If  the  time  for  the  synchronous  online  course  does  not  fit  their  schedule,  they  would  not  take  it. 

Conclusion 

Although  it  is  still  ongoing,  this  study  suggests  that  there  are  not  many  differences  in  terms  of  the  way  the 
participants  learn  and  prepare  for  class  between  traditional  face  to  face  courses  and  synchronous  online  courses.  The 
participants  still  need  to  spend  the  same  amount  of  effort  to  work  on  the  assignments,  as  well  as  attend  the  course 
discussion  online  at  the  specific  time  block  One  difference  is  that  the  participants  are  more  physically  relaxed  in  the 
synchronous  online  course.  One  disadvantage  for  these  Asian  participants  in  the  synchronous  online  course  is  that 
they  cannot  listen  to  the  lecture  and  read  the  instant  messages  at  the  same  time.  They  have  to  concentrate  on  one  or 
the  other  of  them. 

In  this  study,  several  patterns  regarding  the  participants’  views  and  learning  experiences  emerged.  Each 
pattern  represents  a concern  of  the  participants  and  also  provides  the  answers  to  our  research  questions  regarding 
Asian  students’  communication  in  the  synchronous  online  course  as  well  as  their  opinions  of  it.  These  findings  can 
be  summarized  as  follows: 

■ Synchronous  online  courses  are  useful,  but  they  cannot  substitute  for  face  to  face  courses 

■ Synchronous  online  courses  lack  the  sense  of  learning  community,  and  lack  the  informal  interactions  with 
classmates 

■ Asian  students  are  nervous  of  speaking  out  in  the  synchronous  online  course  because  they  are  afraid  of 
making  grammatical  mistakes 

■ Language  barriers  are  the  biggest  challenge  for  Asian  students  in  synchronous  online  courses 

■ Asking  questions  and  expressing  ideas  is  more  difficult  in  synchronous  online  courses 

In  addition,  several  ways  have  been  identified  to  improve  Asian  students’  learning  experience  in  the 
synchronous  online  course: 

■ Establish  some  regular  face  to  face  meetings  for  synchronous  online  courses 

■ Speak  slower  in  synchronous  online  courses  (both  the  instructor  and  students) 

■ Incorporate  some  asynchronous  techniques  to  enhance  learning  in  synchronous  online  courses 

■ Design  some  inter-group  activities  to  encourage  interactions  in  synchronous  online  courses 

■ Summarize  course  discussions  at  the  end  of  class 
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Future  studies 

This  study  has  provided  a clearer  picture  of  taking  synchronous  online  courses  in  the  United  States  from  the 

perspectives  of  Asian  students.  Some  potential  topics  for  further  research  have  emerged  from  this  study.  They  are: 

■ Identify  the  asynchronous  online  techniques  which  can  help  Asian  students  to  express  their  ideas,  as  well 
as  how  to  incorporate  these  components  into  synchronous  online  courses. 

■ How  to  manage  two  way  communications  in  synchronous  online  courses? 

■ How  to  design  group  and  inter-group  activities  to  help  Asian  students  get  involved  in  course  discussions? 

■ How  to  balance  online  and  face  to  face  components  in  a course  to  help  Asian  students  learn  better? 
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Abstract 

Recently , attempts  have  been  made  to  use  Weblog  and  other  personal  webpublishing  technologies  to 
support  individual  and  social  reflection  in  higher  education.  Due  to  Weblog 's  highly  individual  and  reflective 
nature,  students’  experiences  and  perceptions  of  the  technology  and  practice  is  of  primary  importance  in  furthering 
its  educational  use.  In  this  phenomenological  study  nine  participants,  who  maintained  Weblogs  in  a graduate 
course,  were  interx’iewed.  Initial  data  analysis  indicates  that  participants  found  Weblogs  helpful  for  learning, 
reflecting,  and  building  a sense  of  community.  However,  participants  expressed  concerns  over  the  lack  of  structure 
for  Weblog  usage  and  the  public  nature  of  the  reflective  process. 

Introduction 

Reflective  learning  and  thinking  is  widely  regarded  as  being  important  within  all  contexts,  especially,  in 
higher  education  (Brockbank  & McGill,  1998).  Reflective  thinking  was  originally  defined  by  Dewey  (1933)  as 
purposeful  thinking  oriented  toward  a goal.  Reflection  in  learning  involves  a process  of  recording  experience  and 
then  revisiting  the  description  with  the  intent  of  refining  learning  processes  and  strategies  (Boud,  Keogh,  & Walker, 
1985).  Two  propositions  are  important  in  this  context:  one,  that  students  find  it  difficult  to  engage  in  reflection  over 
time  without  external  support  (Harri-Augstein  & Thomas,  1991)  and  two,  that  the  learner  is  primarily  responsible 
for  identifying  important  experiences  and  then  linking  experiences  to  learning  (Boud  et  ah,  1985).  Reflection  is 
governed  by  both  cognitive  and  affective  components  and  the  reflective  act  is  preceded  and  accompanied  by 
individual  feelings  and  perceptions  that  could  either  form  barriers  to  or  enhance  learning.  Understanding  the 
affective  components  of  reflection  and  their  effect  on  student  learning  and  experiences  is  essential  for  identifying 
appropriate  uses  of  reflection  to  assist  individual  learning.  In  this  paper,  we  identify  and  explore  student  feelings  and 
their  lived  experiences  of  reflecting  on  a Weblog  (which  is  a technology  supported  instantiation  of  a reflective 
journal). 


The  Role  of  Reflection  in  Learning 

Within  a constructivist  philosophy  of  learning,  learners  are  the  determinants  of  the  structure  and  sequence 
of  their  knowledge  construction.  Ultimately,  the  learner  is  responsible  for  his  or  her  own  learning  and  the  teacher 
acts  as  a facilitator  to  guide  the  students  learning  (Moon,  1999).  From  a cognitive  perspective,  learning  is  made 
more  meaningful  when  it  is  integrated — that  is  assimilated  and  accommodated — within  the  learner’s  prior 
knowledge  (Piaget,  1970).  Meaningful  learning  is  also  guided  by  experience  and  interpretation  of  experience. 
Learning  can  be  viewed  as  a process  of  revising  the  interpretation  of  an  experience,  which  leads  to  new  ways  of 
thinking,  valuing,  and  acting  (Mezirow,  1990).  The  importance  of  reflection  in  engendering  meaningful  learning  is 
addressed  by  Kolb  (1976),  who  proposed  a cyclical  four  stage  model  where  concrete  experiences  serve  as  the  basis 
for  observation  and  reflection,  which  in  turn  allows  for  abstraction  and  application  in  new  situations.  The  British 
Further  Education  Curriculum  and  Development  Unit  (FEU)  proposed  another  model  of  learning  where  reflection 
occupies  a central  place  in  learning:  specific  reflective  activities  are  vital  for  extracting  specific  learning  of  skills  and 
knowledge  from  an  experience  (FEU,  1981). 

The  earlier  models  clarify  cognitive  components  of  reflection:  affect  is  another  important  influence  on  the 
reflective  act  (Boud  et  al.,  1985).  Current  and  previous  experiences  affect  the  perceptions  of  the  learner  during  the 
reflective  act:  positive  experiences  can  enhance  learning,  while  negative  experiences  can  form  barriers  to  learning 
(Boud  et  al.,  1985).  Kelly  (1955),  as  part  of  his  work  in  personal  construct  psychology,  suggested  that  objects, 
events,  and  constructs  are  meaningful  only  when  viewed  from  the  perspective  of  the  person  construing  the  meaning. 
In  the  context  of  reflection,  Boud,  Keogh  and  Walker  (1985)  identify  the  importance  of  understanding  learner 
intentions  and  perceptions  regarding  a specific  reflective  activity  or  purpose.  Mezirow  ( 1990)  identifies  the 
importance  of  understanding  learners’  feelings  about  their  perceptions,  thinking,  action,  or  habits  of  engaging  in 
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reflection — that  is,  understanding  learners’  affective  reflectivity. 


Engaging  Students  in  Reflection 

Numerous  strategies  have  been  recommended  for  encouraging  reflection,  such  as  Socratic  questioning, 
journal  writing.  Interpersonal  Process  Recall  (IPR),  and  reflecting  teams  (Griffith  & Frieden,  2000).  Journal  writing 
serves  a number  of  purposes,  including  allowing  a student  to  externalize  his  or  her  reflections  on  experiences 
(Shekel  & Trimmer,  1994)  and  then  to  reframe  experiences  within  the  current  context  (Andrusyzyn  & Davie,  1997). 
Writing  about  their  experiences  may  allow  students  to  “think  critically  and  develop  keener  insights  into  assumptions 
and  beliefs  that  can  interfere  with  their  judgments”  (Griffith  & Frieden,  2000).  For  example,  Jasper  (1999)  suggested 
that  journaling  might  provide  a mechanism  for  nurses  to  develop  analytical  and  critical  skills.  Hettich  (1990)  also 
suggested  that  students  liked  journal  writing  because  it  permitted  instructors  to  “monitor”  students’  development  and 
journals  help  them  form  connection  between  concepts  in  class  and  everyday  experiences. 

One  technology-based  instantiation  of  journaling  has  emerged  in  the  form  of  tools  and  practices 
surrounding  personal  Webpublishing.  The  most  common  instantiation  of  personal  Webpublishing  is  a Weblog, 
which  offers  mechanisms  for  learners  to  publish  their  thoughts,  commentaries,  and  reflections  in  the  form  of 
individual  posts  on  a Web  page.  The  individual  posts  are  time  stamped,  archived,  and  appear  in  reverse 
chronological  order:  that  is,  the  most  recent  posts  appear  on  the  top  of  the  page  and  older  posts  are  at  the  bottom  of 
the  page.  Weblogs  allow  for  personalization  and  customization  by  individual  Webloggers.  Weblogs  offer  a 
relatively  convenient  and  easy  mechanism  for  students  to  journal  their  learning  processes,  and  if  used  appropriately 
this  technology  has  the  potential  of  facilitating  reflective  learning.  Weblogs  support  the  ability  of  students  to  record 
and  revisit  experience,  which  is  an  important  part  of  reflective  learning  (Boud  et  ah,  1985).  Bateson  (1973) 
conceptualized  that  reflection  was  a function  of  distance,  “they  (processes  of  reflection)  are  preconditioned  by 
distance”  (Sorensen,  2004).  In  Bateson’s  learning  theory,  when  there  is  a direct  relationship  between  the  learner  and 
the  object  (to  be  learned),  no  reflection  happens  at  this  point.  However,  when  there  is  an  indirect  relationship 
(distance)  between  the  learner  and  the  to  be  learned,  “the  learner  uses  reflection  as  one  of  the  means  in  his/her 
learning  process... there  is  a systematic  reflection  on  how  to  solve  a problem,  and  the  learner  is  conscious  about  the 
fact  that  he/she  is  learning.  He/she  is  consciously  tied  to  the  situated  conditions  (in  a wide  sense)  and  actively  using 
what  he/she  has  learned  at  other  (lower)  level”  (Sorensen,  2004).  Based  on  Baseson’s  theory,  Sorensen  (2004) 
concluded  that  virtual  environments  are  conducive  to  reflection.  Weblog  environments  can  promote  reflective 
activities  in  learners  since  they  lengthen  the  distance  between  learners  and  the  content  to  be  learned. 

Reflective  journaling  stands  in  contrast  to  more  common  forms  of  online  discussion  or  conversation  such  as 
bulletin  boards  or  chats  even  if  they  all  publish  individuals’  thoughts  and  arguments.  They  differ  in  that  postings  in 
discussions  or  conferences  are  interwoven  and  build  on  each  other,  while  Weblogs  are  more  stand-alone  and 
individualized.  Moreover,  course-based  discussion  boards  are  usually  open  to  a limited  number  of  people  while 
Weblogs  are,  by  design,  available  to  anybody  on  the  Web  and  therefore  open  to  a broader  audience.  The  use  of 
Weblogs  in  education  is  relatively  recent  and  our  understanding  of  the  impact  of  this  technology  in  supporting 
reflection  is  incomplete. 


Weblogs  and  Reflection 

With  the  relatively  recent  introduction  of  Weblogs,  very  little  empirical  research  has  been  reported 
regarding  their  use  in  supporting  different  types  of  thinking  and  activities  in  higher  education.  One  area  of  research 
deals  with  students  perceptions  of  sharing  their  online  journaling  practices.  For  example.  Weblogs  can  be  structured 
to  encourage  sharing  of  students’  journals  by  indexing  all  students’  Weblogs  on  the  same  page.  In  this  sense,  a 
Weblog  bears  some  resemblance  to  a “team  journal”  --  where  a group  of  students  collaboratively  write  a common 
journal  for  the  team-  (Andrusyszyn  & Davie,  1997;  Graybeal,  1987).  Team  journals  reportedly  helped  students  to 
make  sense  of  themselves  and  the  world  around  themselves  through  a “cooperative  shared  venture”  (Andrusyszyn  & 
Davie,  1997).  Another  focus  of  exploration  has  been  the  impact  of  Weblogs  on  interaction  between  students  and 
instructors.  Grennan  (1989)  studied  the  effects  of  sharing  journals  of  graduate  students  with  their  instructors.  He 
reported  that  the  use  of  a personal  tone  increased  the  “warmth  of  an  academic  environment”.  It  was  perceived  that 
the  sharing  of  journals  narrowed  the  distance  between  students  and  teachers  and  offered  a form  of  security 
(Andrusyszyn  & Davie,  1997). 

However,  there  is  very  little  reported  exploration  into  learners’  individual  perceptions  and  experiences  with 
the  use  of  Weblogs  for  reflection.  As  presented  earlier,  reflection  is  a combination  of  attitudes  and  skills,  of  affective 
and  cognitive  components,  and  we  believe  that  an  understanding  of  students’  experiences  with  and  their  attitudes 
towards  this  technology  and  practice  would  offer  guidelines  or  suggestions  about  how  Weblogs  could  be  used  best 
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to  enrich  students’  learning  experiences.  We  hope  that  the  outcomes  of  this  study  would  serve  as  a springboard  for 
later  research  about  appropriate  uses  of  Weblogs  to  support  individual  reflection  and  group  discourse  in  educational 
settings.  The  research  question  for  this  study  was:  how  do  graduate  students  engaged  in  a course-specific  web 
logging  activities  describe  their  experiences  of  reflecting  on  the  Weblogs  and  what  meanings  do  they  attach  to  those 
descriptions? 


Research  Design 

Field  research  methods  were  employed  because  this  study  aimed  to  explore  the  lived  experiences  of 
graduate  students  in  their  Weblogging  activities.  Within  the  family  of  field  research  methods,  phenomenology  was 
selected  as  the  research  method,  since  phenomenology  is  concerned  with  understanding  and  describing  people’s 
lived  experiences  with  a specific  phenomenon  (Van  Manen,  1997).  The  current  study  is  to  explore  graduate 
students’  lived  experiences  of  Web-logging.  Also,  phenomenology  aims  to  gain  an  in-depth  understanding  about  the 
phenomenon  at  issue  by  searching  for  its  essence  and  invariant  structure  (Van  Manen,  1997).  Likewise,  the  current 
study  wishes  to  address  the  nature  of  course-specific  Weblogging  phenomenon. 

Participants 

Participants  of  the  study  were  nine  doctoral  students  —four  males  and  five  females  between  24  and  40  years 
old—  who  enrolled  in  one  graduate  level  course  offered  at  northeastern  land-grant  university.  The  study  participants 
maintained  Weblogs  for  one  academic  semester,  as  part  of  a graduate  course  requirement.  Criterion  sampling 
(Creswell,  1998)  was  used  to  select  participants  from  the  classes  who  had  maintained  Weblogs  for  at  least  one 
semester.  In  addition,  the  subjects  were  “willing  to  participate  in  a lengthy  interview,  and  grant  the  investigator  the 
right  to  tape-record’’  (Moustakas,  1994). 

Data  collection 

An  interview  protocol  with  several  open-ended  questions  was  designed  by  referring  to  the  guidelines 
developed  by  Moustakas  (1994)  and  examples  provided  by  Van  Manen  (1997)  and  Creswell  (1998).  An  “interview 
guide  approach’’  (Rossman  & Rallis,  2003)  was  used  to  conduct  one-hour  interviews  with  each  individual 
participant.  Interview  questions  revolved  around  the  following  main  themes—  students’  feelings  toward  using 
Weblogs  in  class,  their  approach  to  reflecting  on  the  Weblogs,  and  their  perceptions  of  its  utility  in  their  learning. 

Data  triangulation 

A second  interview  of  about  20  minutes  will  be  conducted  to  follow  up  with  additional  questions  and  conduct 
member  checks.  The  participant  Weblogs  will  also  be  analyzed  to  provide  an  additional  level  of  triangulation. 

Data  analysis 

All  interviews  were  transcribed  and  N-Vivo  was  used  to  analyze  data.  Free  codes  and  axial  codes  were 
created  based  on  the  analyses  of  these  interview  transcripts.  Preliminary  categories  of  students’  lived  experiences  of 
using  Weblogs  in  a class  were  found  and  are  presented  in  the  following  section.  Data  analysis  comprised  a 
preliminary  grouping  (also  called  “horizonalization’’  (Moustakas,  1994)),  thematic  labeling,  and  finally  developing  a 
textual-structural  description.  The  aim  was  to  develop  a composite  description  of  the  meanings  and  essences  of  the 
experience  for  the  whole  group  of  participants  and  identify  different  influences  on  perceptions  and  use  of  the 
Weblogs  in  contributing  to  students’  reflective  thinking  and  learning. 

Initial  Themes  and  Findings 

Data  analysis  is  still  ongoing,  but  we  present  here  some  of  the  initial  themes  that  have  emerged  through 
analysis.  Overall,  student  experiences  can  be  grouped  into  positive  and  negative  (or  hesitant)  feelings  about  the  use 
of  Weblogs.  Most  students  felt  positively  towards  their  Weblogging  experiences  because  they  felt  that  Weblogging 
helped  learning  and  thinking  and  the  environment  offered  a space  outside  of  the  class  where  they  could  “meet’’  and 
discuss  so  that  a sense  of  community  was  created.  Being  graduate  students  in  instructional  systems  design,  they  were 
excited  to  be  able  to  explore  a new  kind  of  technology  themselves  and  to  understand  and  tap  its  potential  for  their 
own  future  instructional  design. 
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Positive  Perspectives  about  Weblog  Usage 

Blogging  helped  thinking  and  learning 

The  first  theme  in  the  positive  experiences  of  Weblogging  speaks  to  the  utility  of  Weblogs  in  aiding 
thinking  and  learning.  Primarily,  participants  thought  that  Weblogs  supported  their  learning  by  providing  different 
viewpoints,  by  providing  a space  where  reflection  and  commentary  could  be  organized  and  whereby  changes  in 
thinking  could  be  charted,  and  by  pushing  them  to  think  more  critically  about  reading  materials  and  resources. 

Participants  suggested  that  the  instructor’s  and  other  students’  Weblogs  were  useful  for  their  learning  and 
thinking  because  they  offered  different  points  of  view  and  additional  information  beyond  the  class  content,  and 
connected  the  learning  content  to  their  own  experiences. 

“. . .have  others  to  offer  their  thoughts  and  opinions  . . . and  offer  another  point  of  view,  something  else  from 
another  point  of  view.  And  now  you  are  looking  at  the  whole  spectrum,  you  are  not  looking  at  one  part  of  a 
circle.  In  fact,  you  are  looking  at  many  arcs  in  hopes  of  completing  a circle  of  knowledge  toward  a 
particular  subject”  (Interviewee  2,  Paragraph  64). 

So  if  somebody  else  likes  to  contribute  their  experience,  I thought  that  enriched  the  course,  maybe 
challenged  some  views  I thought  the  answers  have  to  be  this  way.  But  then  somebody  else  does  some  other 
way  and  they  get  better  results.  So  I let  me  consider  other  alternatives  (Interviewee  3,  Paragraph  75). 

The  Weblog  provided  a space  where  students  could  organize  their  thinking  and  synthesize  their  learning 
and  students  appeared  to  find  it  useful  to  have  a structured  space  where  they  could  construct  their  learning. 

“...  it  gave  me  a space  I can  summarize  my  thoughts  ...  I think  it’s  good  students  have  opportunities  to  sit 
down  and  write  something,  esp.  reflective  thinking  because  writing  is  a process  that  makes  your  thinking  in 
a ordered. ..’’(Interviewee  7,  Paragraph  138). 

“you  take  learning,  bring  it  into  your  own  context  and  structure  it  to  make  more  sense.  The  process  I was 
going  through  that  I thought  was  rewarding.  I was  able  to  construct  these  blocks  into  new  blocks  of  my 
own.  When  I came  into  the  class  with  my  own  mental  model  of  how  the  pieces  work  together,  instead  of 
just  pieces...”  (Interviewee  3,  Paragraph  95) 

Weblogs  seemed  to  indicate  the  ability  of  “charting”  students’  changes.  As  students  read  their  own  blogs, 
they  could  see  their  own  changes  and  growth  in  thinking.  Being  able  to  see  those  changes  allowed  them  to  better 
reflect  on  their  learning. 

“. . .Weblogs  can  provide  a discourse  for  reflection.  ...  It  sort  of  lays  out  for  you  a roadmap  of  your 
development  in  an  area...”  (Interviewee  2,  Paragraph  156) 

“. . .one  of  the  things  I did  through  Weblogs  was  to  reflect  on  our  personal  growth  in  terms  of  instructional 
design  ideas  and  so  on.  So  that  writing  Weblogs  help  me  articulate  our  ideas  and  also  I could  compare 
ideas.  That  was  what  Weblogs  are  meant  to  be  because  like  a history  of  my  thoughts.  So  I could  see 
changing.  Maybe  if  I haven’t  written  those  things,  even  those  changes  might  happen,  but  when  I actually 
wrote  them  down,  it  probably  increased  the  pace  those  changes  were  happening.”  (Interviewee  5,  Paragraph 
8) 

It  appeared  that  Weblog  publishing  encouraged  students  to  adopt  deep  approaches  to  learning.  In  order  to 
find  something  to  write  about  in  Weblogs,  students  had  to  focus  their  attention  on  their  reading  and  were  “pushed 
into  deep  thinking”  about  the  content. 

“...  for  me  as  a student,  it  was  an  opportunity  to  do  something  with  the  material  aside  from  just  reading  it. 
So  it  was  educational  experience  just  writing  itself. . . because  it  keeps  me  actively  interested  in  what  I am 
reading  instead  of  sometimes  when  you  read,  you  drift  off,  you  lost  your  attention.  But  it  keeps  my 
attention.”  (Interviewee  3,  Paragraph  68) 

“...  it  helped  me  to  look  at  the  content  even  more  questioning.  Like  I have  to  find  questions  here  ‘cause  I 
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need  to  write  something.  I want  to  write  something,  I need  to  write  something,  I must  find  the  question.  I 
would  look  possibly  more  critically  or  try  to  find  places  where  I would  want  to  know  something  more  or 
where  didn’t  fit  my  experiences.”  (Interviewee  4,  Paragraph  68) 

Blogging  offered  a sense  of  community 

A second  positive  theme  was  the  role  of  blogging  in  creating  a sense  of  learning  community  and  extended 
discussion  in  and  outside  of  the  class. 

“My  feeling  is  that  you  are  part  of  a community.  That  was  nice.  You  feel  you  are  part  of  a group,  part  of  an 
effort.  You  are  actually  included  something.  You  don’t  feel  you  are  alone,  because  even  though  you  are 
new  to  the  field,  you  are  still  struggling.  And  it  is  that  feeling  of  cooperate  that  we  are  all  struggling  to 
understand  this  together  ...  At  least  it  opened  up  that  avenue  that  I wasn’t  going  through,  trying  to 
understand  it  alone.”  (Interviewee  2,  Paragraph  145) 

“. . . (the  instructor)  is  like  connect  other  people  together.  She  summarized  what  other  people  talked 
about.  ...  it  is  good  because  we  normally  don’t  talk  about  other  people,  but  she  talked  about  other  people’s 
log  too  so  we  also  know  what’s  going  on.  It  brings  a feeling  that  we  now  know  all.  We  all  belong  to 
something,  something  we  can  interact  with  each  other.”  (Interviewee  9,  Paragraph  68) 

“Sometimes  reading  what  they  had  written  and  when  I see  them  in  person  face  to  face,  that  would  generate 
a conversation.  Saying,  “hey,  that  thing  you  wrote  on  Wednesday,  that  was  very  interesting”,  or  maybe  I 
have  a counter  point  or  ask  them  more  about  what  happened  to  get  more  details.  So  maybe  that’s  additional 
thing  it  did,  was  to  open  a door  for  additional  communication  face  to  face.”  (Interviewee  4,  Paragraph  18) 

Blogging  offered  exploration  of  new  technologies 

Students  were  excited  to  be  able  to  explore  a new  technology.  Since  most  of  the  students  were  graduate 
students  in  instructional  systems  design,  the  Weblogging  experience  itself  made  the  students  excited  in  that  it 
provided  them  a hands-on  opportunity  to  fully  explore  an  instructional  technology. 

“Because  I never  used  Weblog  before,  it’s  been  interesting.  I want  to  use  it  to  see  how  it  is.  I was  pretty 
excited  about  using  it  because  I never  heard  of  it.  Actually,  I think  the  instructor  told  me  that  it  has  been 
pretty  widely  used  for  many  other  people  but  it’s  my  first  exposure  to  Weblog.”  (Interviewee  9,  Paragraph 

5) 


“...  since  the  course  is  an  introductory  course,  most  of  the  students  have  limited  experience  with 
instructional  design.  Then  when  we  were  introduced  of  this  website,  I think  it’s  like  something  new.  The 
Weblog  provides  space  for  you  to  write  journals,  I think  that’s  good.”  (Interviewee  7,  Paragraph  6) 


Hesitancy  or  Negative  Perceptions  about  Weblog  Usage 

Apart  from  the  positive  feelings  associated  with  the  use  of  Weblogs,  participants  also  expressed  some 
hesitancies.  For  many,  it  was  their  first  introduction  to  such  a technology,  and  they  experienced  cognitive 
difficulties,  which  in  turn  resulted  in  stressful  experiences  for  some  students.  Because  of  its  highly  individual  and 
open  nature,  students  had  concerns  about  the  environment,  people’s  backgrounds,  and  the  topics  they  chose  to 
discuss.  Main  themes  were  related  to  uncertainty  on  Weblog  usage  and  the  need  for  more  structure  and  guidance  on 
their  use. 

Uncertainty  about  correct  usage  of  Weblogs 

Because  most  of  the  students  had  never  used  Weblogs  before,  they  mentioned  their  uncertainty  about  the 
usage  of  the  technology.  The  fact  that  the  Weblogs  were  a part  of  class  requirement  and  would  be  graded  at  the  end 
of  the  class  exerted  some  amount  of  pressure  on  these  students. 

“I  think  there  was  pressure  since  it  was  the  first  time  I was  doing  it.  I was  new  to  it.  So  there  was  pressure 
of  doing  it  right  and  correct.  Because  it  was  new,  I started  slow  and  that  kind  of  put  some  pressure.  That 
kind  of  affected  the  course  of  my  Weblogging.  I wasn’t  doing  it  as  early  as  I should.”  (Interviewee  5, 
Paragraph  42,  43) 
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Being  exposed  to  Weblogs  for  the  first  time,  some  students  had  not  yet  understood  how  they  could  use 
Weblogs  in  their  learning  and  thinking.  Thus,  they  engaged  in  the  activity  for  only  the  purpose  of  meeting  the  class 
requirement.  For  such  students.  Weblogging  became  a somewhat  stressful  experience. 

“They  can  feel  this  external  pressure  to  make  these  postings.  And  they  just  post  something  just  meet  the 
assignment  but  not  reflecting,  or  they  don’t  really  feel  comfortable . . . (Interviewee  3,  Paragraph  68) 

“...  if  I have  something  interesting  periodically,  I can  write  something  in  the  Weblog.  But  during  a 
semester  you  are  very  busy  with  other  courses,  so  it’s  very  stressful  for  me  to  figure  out  something  to  write 
there’’  (Interviewee  8,  Paragraph  10) 

Since  the  students  had  no  idea  of  how  to  blog  at  the  beginning  of  the  class,  they  were  expecting  more 
“structures  or  guidelines’’  from  the  instructor  about  how  to  blog  and  what  to  blog. 

“(We  should  be  told)  “you  should  post  a paper  about  this  topic,  you  should  write  about...”  It’s  a more 
structural  way . . . One  day  you  can  write  about  schedule  of  writing,  one  day  you  can  write  about  article, 
then  you  can  write  about  discussion,  then  people  will  have  an  idea  about  how  to  post  Weblog.” 

(Interviewee  9,  Paragraph  111) 

In  absence  of  such  guidelines  and  exact  directions,  some  students  felt  lost  and  frustrated. 

“There  was  no  definitive,  “this  is  what  you  have  to  do,  I want  you  to  write  about  this  topic  or  tell  me  your 
thoughts  on  this  in  particular”.  It  was  so  open  that  when  you  have  no  limitations,  it’s  hard  to  figure  out 
what  you  write,  what  should  be  public  knowledge,  and  what  you  keep  private,  and  what  the  instructor  feels 
is  meaningful  as  compared  to  what  is  nonsense.  It  was  hard  because  there  were  no  guidelines.  So  it  was 
frustrating  in  that  sense”.  (Interviewee  6,  Paragraph  9) 

Role  of  prior  experiences 

Students  found  their  own  backgrounds  mattered  in  their  practice  of  and  ability  to  engage  in  blogging.  The 
students  in  the  class  came  from  very  different  backgrounds  and  their  levels  of  expertise  with  the  course  content 
varied.  Students  with  relatively  more  experiences  with  the  content  were  generally  more  active  with  Weblogging. 
However,  those  who  were  new  to  the  field  found  it  hard  to  create  a topic  or  connect  to  their  prior  experiences  as  the 
more  advanced  students  did. 

“...  at  that  time,  I don’t  have  enough  information  or  knowledge  with  me,  so  it’s  very  hard  to  find  something 
to  write  there.  Because  I am  a just  a first  semester  student,  it’s  very  difficult  to  write  some  topic  in  there.” 
(Interviewee  8,  Paragraph  42) 

The  big  differences  in  students’  background  posed  some  difficulties  for  some  students  too  because  when 
the  other  people  were  writing  or  connecting  with  a totally  unfamiliar  context,  they  found  such  the  content  of  such 
writing  difficult  to  understand  and  apply  to  their  own  situation. 

“...  the  gentleman  I worked  with  was  military  based,  so  all  of  his  analogies  were  based  on  military.  And  I 
am  not  a military  person.  I don’t  necessarily  understand  military  speak,  so  some  of  the  things  that  he  made 
correlations  to  I don’t  understand  at  all  because  it’s  not  even  in  the  realm  of  experience  or  understanding.” 
(Interviewee  6,  Paragraph  59) 

Concerns  about  privacy 

One  student  expressed  doubts  about  the  environment  because  of  its  potential  to  be  viewed  by  anybody  in 
the  world.  Theoretically,  anyone,  anywhere  could  respond  to  the  writing  on  the  Weblogs,  and  the  student  was 
particularly  afraid  of  attacks  on  people’s  ideas  without  proper  reasoning  and  factual  backup. 

“Those  who  would  offer  “insights”,  I put  that  in  quotes,  that  was  not  founded  upon  anything  as  you  can  tell 
is  almost,  in  one  sense,  reaction  of  deeply  held  personal  beliefs  which  are  fine  as  long  as  they  would  have 
the  opportunity  to  reflect  upon  themselves,  inspired  by  research,  not  just  something  you  think  is  right  or 
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wrong  but  why  do  you  think  is  right  or  wrong.  You  disagree  with  me,  that’s  fine,  but  tell  me  why.  Just 
don’t  attach  something  and  offer  no  reason  as  to  your  justification.”  (Interviewee  2,  Paragraph  66) 

The  student  was  also  concerned  about  the  attention  to  be  given  to  responses  by  unknown  individuals,  since  he 
found  it  difficult  to  trust  the  sources,  not  knowing  the  background  or  expertise  of  the  person. 

“...  if  you  don’t  know  the  person  via  the  Weblog...  to  the  depth  and  credibility  of  their  arguments  or  their 
thought,  then  I was  skeptical  and  I was  skeptical  posting  mine  ‘cause  somebody  who  had  no,  very  limited 
knowledge  would  respond  something  then  I don’t  know  how  much  impact  I should  take  that.”  (Interviewee 
2,  Paragraph  146) 

Because  of  the  public  nature  of  Weblogs,  most  students  expressed  their  concerns  about  the  topic  they  wrote 
about.  They  chose  not  to  blog  anything  personal,  anything  ‘‘too  controversial”  or  too  negative. 

“...  things  I thought  were  going  to  be  personal  or  group  issues  didn’t  go  on  the  Weblog,  so  that  being 
public  forum,  my  thoughts  I wanted  it  to  be  kind  of  profession  orientation.  If  it  was  personal  issues, 
personal  problems,  I chose  to  keep  those.”  (Interviewee  4,  Paragraph  60) 

“I  don’t  want  to  name  names.. . I’m  not  going  to  make  anybody  feel  bad  or  make  it  known  to  everyone  else 
in  the  class  that  he/she  was  not  doing  what  he’s  supposed  to  do.  So  I don’t  think  it’s  the  right  place  to  do 
that.  You  know  it’s  the  same  thing  you  air  your  dirty  laundry  in  public.  I don’t  think  it’s  to  be  used  as  a 
sounding  board  for  negativity  in  that  sense...”  (Interviewee  6,  Paragraph  34) 

Future  steps 

Data  analysis  is  still  on-going  and  we  intend  to  integrate  and  structure  the  emergent  themes.  Other  primary 
themes  were  found  but  are  still  in  need  of  further  refinement  and  exploration.  We  present  some  of  these  other 
themes  in  overview  as  a precursor  to  more  detailed  forthcoming  descriptions.  These  themes  include: 

a.  students’  perceived  comparison  between  Weblog  and  integrated  learning  environment  such  as  WebCT, 
online 

b.  discussion  board,  and  listserv  postings  etc. 

c.  students’  points  of  view  about  the  technical  aspect  of  the  Weblogging  environment 

d.  students’  Weblog-visiting  and  -responding  patterns 

e.  students’  interpreted  purpose  and  their  own  usage  of  Weblogs 

f.  different  modes  of  students’  demonstrated  motivation  types  in  relation  to  their  emotional  experiences  of 
using  Weblogs 

g.  manifested  learning  curve  of  students’  use  of  Weblog 

Conclusion  and  Implications 

As  a new  and  potentially  powerful  technology,  the  parameters  and  consequences  of  Weblog  usage  within 
higher  education  are  still  unexplored.  Understanding  graduate  students’  lived  experience  of  their  web  logging 
activities  has  significant  meaning  for  teachers,  practitioners,  and  instructional  designers  to  use  Weblogs  in 
classrooms.  Especially,  if  Weblogs  are  to  be  used  to  support  specific  types  of  thinking  or  collaborative  practices  in 
educational  settings,  students’  perceptions  are  paramount  in  informing  both  researchers  and  practitioners  in  further 
educational  and  exploratory  implementations.  We  have  presented  some  initial  themes  regarding  the  positive  and 
negative  lived  experiences  of  students’  Weblogging  activities.  This  initial  data  supports  the  proposition  that 
Weblogs  can  be  used  to  support  reflection  individually;  however,  data  also  suggest  that  a more  structured  and 
guided  introduction  to  the  usage  of  this  technology  would  be  conducive  to  inculcating  early  and  appropriate  usage. 
Another  initial  theme  suggests  that  Weblogs  allowed  students  to  structure  and  chart  their  learning  progress.  Building 
on  this  initial  finding,  it  would  be  useful  to  identify  additional  aids  to  help  students  build  and  further  structure  their 
learning  and  reflective  activities.  For  example,  a combination  of  Weblogs  and  concept  mapping  activities  might 
allow  students  to  map  their  learning  in  a more  concrete  fashion,  thereby  linking  individual  posts  into  a more  holistic 
picture  of  content  learning.  Although  it  appeared  that  Weblogs  could  potentially  offer  a sense  of  community,  the 
threat  of  privacy  invasion  also  loomed  large  for  some  students.  Further  exploration  is  need  to  achieve  a balance  in 
fostering  community,  while  avoiding  situations  that  students  might  find  threatening  or  disengaging. 
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Abstract 

With  a growing  number  of  courses  offered  online  and  degrees  offered  through  the  Internet,  there  is  a 
considerable  interest  in  online  education,  particularly  as  it  relates  to  the  quality  of  online  learning  instruction.  The 
major  concerns  are  centering  on  the  following  questions:  What  will  be  the  new  role  for  instructor  in  online 
education?  How  will  students’  learning  outcomes  be  assured  and  improved  in  online  learning  environment?  How 
will  effective  communication  and  interaction  be  established  with  students  in  the  absence  of  face-to-face  instruction? 
and  How  will  instructors  motivate  students  to  learn  in  the  online  learning  environment?  This  paper  will  examine 
new  challenges  and  barriers  for  online  instructors,  highlight  major  themes  prevalent  in  the  literature  related  to 
“quality  control  or  assurance”  in  online  education,  and  provide  strategies  for  instructors  to  design  and  deliver 
effective  online  instruction.  Recommendations  will  be  made  on  how  to  ensure  quality  in  online  instruction,  and  the 
role  of  administrators  in  ensuring  quality  online  learning  will  also  be  described. 


With  a growing  number  of  courses  offered  online  and  degrees  offered  through  the  Internet,  there  is  a 
considerable  interest  in  concerns  and  problems  associated  with  online  education,  particularly  as  it  relates  to  the 
quality  of  online  education  instruction  (Allen  & Seaman,  2003).  According  to  Twigg  (2001)  many  problems  that 
arise  from  online  education  as  it  relates  to  quality  include,  but  is  not  limited  to:  (a)  the  requirement  of  separate 
quality  assurance  standards,  (b)  programs  having  low  (or  no)  quality  standards,  and  (c)  there  being  no  consensus  on 
what  constitutes  learning  quality. 

Online  education,  according  to  Harasim  (1989),  is  a new  domain  of  learning  that  combines  distance 
education  with  the  practice  of  face-to-face  instruction  utilizing  computer- mediated  communication.  Volery  (2000) 
concluded  that  online  delivery  is  a form  of  distributed  learning  enabled  by  the  Internet.  Ascough  (2002)  suggested 
that  online  education  has  the  following  features:  (a)  it  provides  a learning  experience  different  than  in  the  traditional 
classroom  because  learners  are  different,  (b)  the  communication  is  via  computer  and  World  Wide  Web,  (c) 
participation  in  classroom  by  learners  are  different,  (d)  the  social  dynamic  of  the  learning  environment  is  changed, 
and  (e)  discrimination  and  prejudice  is  minimized.  More  recently,  Allen  and  Seaman  (2003)  in  conducting  a survey 
on  online  education  delivered  by  higher  education  institutions  in  the  United  States  defined  an  online  course  as  one 
that  had  at  least  80  % of  the  course  content  delivered  online.  Regardless  of  the  definition,  an  early  indication  of  the 
widespread  popularity  of  online  education  courses  can  be  found  in  a survey  conducted  by  the  U.S.  Department  of 
Education,  which  revealed  that  more  than  54,000  online  education  courses  were  being  offered  in  1998,  with  over  1.6 
million  student’s  enrolled  (cited  in  Lewis,  et  al.,  1999).  In  this  study,  Allen  and  Seaman  (2003)  reported  that:  (a) 
over  1.6  million  students  took  at  least  one  online  course  during  the  Fall  of  2002,  (b)  over  one-third  of  these  students 
(578,000)  took  all  of  their  courses  online,  (c)  among  all  U.S.  higher  education  students  in  Fall  2002,  1 1 percent  took 
at  least  one  online  course,  and  (d)  among  those  students  at  institutions  where  online  courses  were  offered,  13  percent 
took  at  least  one  online  course  (p.l). 


Background  of  the  problem 

Controversies  as  to  the  quality  of  online  education  have  not  diminished  over  the  past  decades.  Many  people 
are  suspicious  of  online  education  because  courses  are  often  offered  by  divisions  of  extended  studies  or  continuing 
education  (Husmann  & Miller,  2003)  and  are  taught  by  adjunct  faculty  or  instructors  who  have  not  earned  doctoral 
degrees.  Therefore,  many  individuals  have  concluded  that  online  education  programs  are  left  outside  of  formal 
faculty  structures  that  have  traditionally  had  oversight  for  instructional  course  quality.  Both  proponents  and 
opponents  have  been  concerned  about  online  education  quality.  Opponents  view  online  education  as  inferior,  see  it 
as  a substitute  for  the  traditional  “brick  and  mortar”  university,  and  conclude  it  is  rather  a profit  making  venue.  This 
type  of  delivery  is  often  viewed  by  “administrators  as  a “cash  cow” -a  means  of  delivering  instruction  to  a large 
number  of  paying  customers  without  the  expense  of  providing  things  such  as  temperature  controlled  classrooms  and 
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parking  spaces”  (Brown  & Green,  2003,  p.  148- 149).  Opponents  have  also  suggested  that  online  courses  lower  the 
quality  of  academic  standards  (Buck,  2001).  Some  opponents  even  question  the  quality  of  online  courses  when 
students  do  not  actually  attend  a college,  and  have  face-to-face  interaction  with  instructors.  Moreover,  Weiger 
(1998)  asserted  that  the  quality  of  instructors  who  teach  online  courses  cannot  be  guaranteed  since  anyone  can  put  a 
course  online. 

Concerns  regarding  the  quality  of  online  education  are  also  raised  by  both  students  and  faculty.  Arguments 
are  made  that  as  consumers  of  online  education,  students  are  unlikely  to  be  able  to  find  out  information  about  the 
quality  of  the  courses  that  are  provided  (Twigg,  2001).  Schools  or  universities  that  offer  online  education  courses 
typically  do  not  provide  comparative  information  for  students  e.g.,  how  would  a student  know  which  online  course 
meets  his/her  needs?  Moreover,  prerequisites  that  are  essential  for  taking  a particular  online  course  are  usually  not 
clearly  stated  on  websites  for  students,  and  “when  students  are  encountering  technical  problems,  whom  they  can  ask 
for  assistance  is  not  available  to  them”  (Twigg,  2001,  p.  15). 

From  the  faculty’s  perspective,  if  they  haven’t  received  the  training  for  teaching  online  courses,  using  the 
technologies,  evaluating  and  assessing  online  courses,  how  then  can  the  quality  of  their  online  teaching  be  assured? 
Moreover,  when  teaching  online,  if  a majority  of  the  faculty  member’s  time  is  spent  corresponding  with  students, 
how  then  can  faculty  balance  their  traditional  teaching,  research,  and  service  activities?  When  faculty  are  reluctant 
to  teach  online  classes,  how  can  school  administrators  to  motivate  them  to  do  so? 

Proponents  are  in  support  of  online  education.  They  suggest  that  the  lack  of  face-to-face  interaction  can  be 
substituted  by  online  discussions  in  bulletin  board  systems,  online  video  conferences  or  on  listservs  (Blake,  2000). 
Online  education  can  also  promote  students’  critical  thinking  skills,  deep  learning,  collaborative  learning,  and 
problem-solving  skills  (Ascough,  2002;  Rosie,  2000).  Donlevy  (2003)  asserted  that  online  education  may  help 
schools  expand  curricula  offerings  with  less  cost  and  can  help  graduates  gain  important  technology  skills  to  improve 
their  marketability.  Proponents  also  argue  that  online  education  can  encourage  non-discriminatory  teaching  and 
learning  practices  since  the  teachers  and  students,  as  well  as  students  and  their  classmates  typically  do  not  meet  face- 
to-face.  Palloff  and  Pratt  (1999)  reported  that  because  students  cannot  tell  the  race,  gender,  physical  characteristics 
of  each  other  and  their  teachers,  online  education  presents  a bias -free  teaching  and  learning  environment  for 
instructors  and  students. 


Quality  Assurance  of  Online  Education 

The  quality  of  online  education  has  prompted  the  attention  of  higher  education  accreditation  associations. 
The  report  of  the  Council  for  Higher  Education  Accreditation  (1998,  as  cited  in  Weiger,  1998)  recommended  that 
accreditators  should  “establish  reliable  and  valid  performance  measurements,  require  evidence  of  contact  between 
faculty  and  students,  mandate  evidence  of  effective  instructional  techniques,  promote  systematic  efforts  to  select  and 
train  faculty,  and  assure  that  students,  faculty,  staff  and  administrators  receive  adequate  training  to  use  electronic 
resources”  (p.  1 1).  Therefore,  the  need  of  standards  for  ensuring  quality  of  online  education  instruction  is 
paramount. 

Paulsen  (2002)  in  defining  online  education  indicated  that  it  separates  teachers  and  learners  (which 
distinguishes  it  from  face-to-face  education),  influences  an  educational  organization  (which  distinguishes  it  from 
self-study  and  private  tutoring),  uses  computer  network  to  present  or  distribute  some  educational  content,  and 
provides  two-way  communication  via  a computer  network  so  that  students  may  benefit  from  communication  with 
each  other,  teachers,  and  staff  (p.  1 .).  This  definition  clarifies  the  difference  between  online  education  and  traditional 
education.  Consequently,  quality  indicators  should  be  different  as  it  relates  to  online  education  and  traditional 
education. 

The  higher  education  community  has  developed  several  quality  indictors  for  traditional  education  that  are 
well  accepted  by  many  institutional  quality  assurance  programs  (Twigg,  2001).  Twigg  (2001)  has  stated,  “For 
traditional  education,  quality  equals  a tenured  full-time  faculty  member  with  a doctoral  degree  teaching  the  course. 
Quality  equals  courses  and  degree  programs  offered  by  and  on  a residential  campus.  Quality  equals  “students 
learning  by  sitting  with  a professor  face-to-face”  (Twigg,  2001,  p.  3).  However,  in  online  education,  students  will 
not  know  if  the  instructor  has  earned  doctoral  degree  or  not,  because  there  is  no  way  to  gain  the  knowledge  of  the 
instructor’s  background  unless  the  instructor  him/ herself  indicated  on  the  course  website.  Online  education  is 
usually  regarded  as  time  saving  and  cost  effective  education  since  students  do  not  need  to  drive  to  a classroom. 
Face-to-face  instruction  is  often  not  guaranteed  in  online  class.  Therefore,  those  common  quality  indictors  do  not 
and  should  not  apply  to  online  education. 

In  the  early  1990s,  the  Western  Cooperative  for  Educational  Telecommunications  (WECT))  developed 
“Principles  of  Good  Practice  for  Electronically  Offered  Academic  Degree  and  Certificate  Programs”  (Twigg,  2001). 
Since  then,  many  other  groups  have  developed  similar  principles  and  practices.  For  example.  The  American 
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Distance  Education  Consortium  (ADEC)  drafted  “ADEC  Guiding  Principles  for  Distance  Learning”.  A joint  task 
force  of  the  American  Council  of  Education  and  the  Alliance:  An  Association  for  Alternative  Programs  for  Adults 
developed  ‘‘Guiding  Principles  for  Distance  Learning  in  a Learning  Society.”  The  Instructional  Telecommunications 
Council  provided  “Quality  Enhancing  Practices  in  Distance  Education.”  The  American  Federation  of  Teachers 
(AFT)  developed  “Distance  Education:  Guidelines  for  Good  Practice.”  The  Council  of  Regional  Accrediting 
Commissions  updated  and  explained  WECT’s  statement,  and  published  “Guidelines  for  the  Evaluation  of 
Electronically  Offered  Degree  and  Certificate  Programs”  (Twigg,  2001). 

In  2000,  The  Institute  for  Higher  Education  Policy  (IHEP)  first  reviewed  all  of  the  existing  principles  or 
guidelines,  and  proposed  24  benchmarks  for  measuring  quality  Internet-based  learning,  which  were  grouped  into 
seven  categories:  (a)  institutional  support,  (b)  course  development,  (c)  teaching/learning,  (d)  course  structure,  (e) 
student  support,  (f)  faculty  support,  and  (g)  evaluation  and  assessment  (IHEP,  2000).  Yeung  (2001)  also  carried  out 
a study  among  academic  staff  in  Hong  Kong  higher  education  institutions  on  their  perceptions  of  a quality  assurance 
model.  He  concluded  that  the  benchmarks  for  quality  assurance  of  web-based  learning  were  considered  important. 
The  institutions  that  participated  in  the  study  attempted  to  incorporate  those  benchmarks  into  their  policies, 
practices,  and  procedures.  Yeung  (2001)  further  asserted  that  academic  staff  and  students  are  the  two  key 
stakeholders  in  an  educational  setting.  Therefore,  to  ensure  the  quality  of  online  education,  controlling  the  quality  of 
academic  faculty  who  teach  online  courses  is  vital. 

The  team  approach  has  also  been  suggested  by  organizations  and  researchers  as  a method  to  ensure  the 
quality  of  online  education  instruction.  The  Southern  Regional  Education  Board  (2001)  encourages  institutions  and 
states  to  build  an  instructional  design  team  for  a quality  online  environment.  Such  a team  might  consist  of  the 
instructional  designer,  graphic/interface  designer,  technical  support  personnel,  content  expert,  direct  instructor, 
information  resource  personnel,  mentors/tutors,  and  assessor.  The  instructor,  however,  remains  at  the  center  of  the 
team  to  guarantee  academic  integrity,  with  the  assistance  from  other  partners.  Levy  (2003)  suggested  an 
organizational  structure  change  in  online  educational  program.  This  change  should  involve  different  people  who  do 
different  jobs.  For  example,  in  this  scenario,  a content  specialist  would  decide  the  teaching  material,  an  instructional 
designer  would  be  responsible  for  the  visual  presentation  of  this  material,  and  a technical  specialist  would  actually 
create  the  online  course  and  the  instructor  then  interacts  with  the  online  learners.  Care  and  Scanlan  (2001)  have  also 
advocated  another  team  approach,  which  is  the  Interdisciplinary  Team  Model.  In  this  model,  various  participants 
meet  as  a team  on  a regular  basis  to  develop  the  course,  solve  problems,  and  discuss  issues  as  course  development 
unfolded.  The  participants  are  content  specialist,  instructional  designer,  student  representative,  media  specialist, 
program  director,  and  external  faculty  member. 

Ensuring  Effective  Online  Instruction 

Challenges  and  Barriers  for  Online  Education  Instructors 

Some  of  the  challenges  and  barriers  for  online  learning  that  have  been  identified  by  researchers  are  the 
change  of  roles  and  responsibilities  for  instructors  (Zheng  & Smaldino,  2003;  Murihead,  2000),  use  of  technology 
(Valentine,  2002;  Palloff  & Pratt,  2000;  Berge,  1998;  Volery,  2000),  interaction  with  students  and  the  changes  in 
interpersonal  relations  (Bower  2001),  and  academic  dishonesty  of  online  learners  (Muirhead,  2000).  O’Quinn  & 
Corry’s  (2002)  in  conducting  a study  on  online  education  pointed  out  several  factors  that  may  deter  faculty  from 
teaching  online.  The  factors  the  authors  identified  include  a lack  of  professional  prestige,  delivery  method  used, 
change  in  faculty  role,  and  lack  of  monetary  support. 

New  roles  of  instructor. 

Instructors  have  many  concerns  about  online  education.  Their  primary  concern  is  how  online  education 
changes  their  roles  and  responsibilities,  and  how  they  can  adapt  to  this  change.  Online  education  is  widely  accepted 
as  student-centered  education,  and  the  traditional  education  is  regarded  as  professor-centered  education.  Due  to  a 
shift  to  online  education,  the  instructor’s  role  has  become  more  of  a facilitator  than  a traditional  lecturer.  Therefore, 
the  traditional  professor-centered  educational  environment  and  student-centered  online  educational  environment  will 
have  many  differences.  Besides  the  role  shifting,  the  role  of  the  virtual  instructor  is  to  select  and  filter  information 
for  student  consideration,  to  provide  thought-provoking  questions,  and  to  facilitate  well-considered  discussion 
(Kettner-Polley,  1999). 

Wu  & Hiltz  (2004)  conducted  a study  of  1 16  students  enrolled  in  two  undergraduate  courses  and  one 
graduate  course  at  the  New  Jersey  Institute  of  Technology.  Their  study  concluded  that  variations  among  instructors 
or  courses  are  associated  with  differences  in  perceptions  of  student  motivation,  enjoyment  and  learning.  Wu  and 
Hiltz  also  found  that  in  traditional  professor-centered  education,  the  roles  of  professor  and  student  are  regimented; 
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the  professor  disseminates  knowledge,  and  the  student  reflects  that  information.  However,  as  Knowton  (2000)  has 
argued,  in  the  student-centered  online  education  course,  the  professor  and  students  are  a community  of  learners.  The 
professor  serves  as  coach,  counselor  and  mentor;  the  students  become  active  participants  in  learning.  During  the 
processes  of  learning,  in  teacher-centered  classroom,  professor  lectures  while  students  take  notes.  In  online  student- 
centered  education,  the  professor  serves  as  facilitator,  while  students  collaborate  with  each  other  and  the  professor  to 
develop  personal  understanding  of  content. 

Murihead  (2000)  indicated  three  areas  considered  to  be  changed  when  the  education  courses  are  put  online: 
(a)  the  provision  of  instructional  and  emotional  support  to  students,  (b)  the  expectations  associated  with  authoring 
online  courses  while  maintaining  a full  teaching  load,  and  (c)  the  requirement  to  provide  ongoing  technological 
support  to  students  and  parents  (p.  322).  According  to  Ascough  (2002),  the  role  of  instructor  in  an  online  learning 
environment  should  be  more  of  a facilitator  or  moderator  due  to  less  control  of  the  class  environment.  He  noted  that 
because  most  instructors  are  more  likely  to  have  been  trained  in  traditional  instruction,  it  is  a somewhat  foreign 
practice  for  them  to  plan  interactive  strategies  in  course  delivery,  and  adjusting  their  change  in  role  from  the  leading 
speaker  to  that  of  a facilitator.  Volery  (2000)  also  suggested  that  the  academic  role  of  instructor  should  be  shifted 
from  intellect-on-stage  and  mentor  towards  a learning  catalyst  because  the  level  of  interaction  has  changed  in  online 
delivery.  Therefore,  besides  being  a facilitator,  the  instructor  should  also  be  an  instructional  designer  (Zheng  & 
Smaldino,  2003). 

New  roles  of  online  learners. 

Because  the  online  environment  is  different  from  the  traditional  classroom,  it  is  important  for  the  instructor 
to  motivate  students  to  adjust  their  roles  when  becoming  an  online  learner.  In  online  education,  the  interaction 
between  students  and  their  instructors  have  been  changed  from  synchronous  in  face  to  face  (F2F)  instruction  to  an 
asynchronous  virtual  community.  Thus,  a significant  role  adjustment  for  students  may  be  required  if  they  are  to 
experience  success.  Students  must  move  from  being  a more  traditional  passive  classroom  learner  into  a more  active 
online  inquirer.  Hughes  (2004)  has  suggested  that  online  learners  should  ask  themselves,  “Am  I ready  for  university 
(or  college)?”  “Am  I ready  for  online  learning?”  “What  is  my  preferred  learning  style?”  “Do  I have  the  skills  to  be 
successful  in  my  chosen  program?”  (p.  369-370). 

Garrison,  Cleveland-Innes,  and  Fung  (2004)  conducted  a study  to  validate  an  instrument  regarding  online 
students’  role  adjustment.  Their  findings  suggest  that  students  do  see  a difference  in  the  learning  process  and  a need 
for  their  role  adjustment.  The  online  learning  should  be  viewed  as  more  cognitive  or  internally  oriented.  Online 
learners  must  take  more  responsibility,  adjust  to  a new  climate,  adjust  to  new  context,  synthesize  ideas,  know  how  to 
participate,  synthesize  ideas,  apply  ideas  or  concepts,  and  stimulate  their  own  curiosity.  In  addition,  Palloff  & Pratt 
(2003)  have  suggested  that  online  learners  should  be  “open”  about  personal  details  of  his  or  her  life,  work  and  other 
educational  experiences;  should  be  “flexible”  and  “humor”  to  create  a warm,  inviting  course  environment;  should 
be  “honest”;  should  be  willing  to  take  “responsibility”  for  online  community  formation;  and  should  be  willing  to 
work  “collaboratively”  (p.  17-28). 

New  technologies 

Technology,  as  the  inter-medium  for  instructors  to  delivery  courses  becomes  more  important  when  the 
level  of  face-to-face  communication  is  decreased  in  online  education  courses.  Consequently,  how  to  appropriately 
use  technology  to  serve  an  instructional  purpose  tends  to  be  another  challenge  for  online  education  instructors. 
Muirhead  (2000)  reported  teachers’  frustration  with  the  reliability  of  computer  technology,  working  with  multiple 
versions  of  a software  package,  providing  technology  support  to  students  using  multiple  operating  systems,  and  the 
absence  of  mature  integrated  content  development  tools.  Palloff  and  Pratt  (2000)  also  noted  that  the  instructor  must 
be  trained  “not  only  to  use  technology,  but  also  to  shift  the  way  in  which  they  organize  and  deliver  material”  (p.  3). 
Valentine  (2002)  indicated  that  misuse  of  technology  could  also  be  a problem  for  the  instructor,  although  this 
problem  may  arise  from  lack  of  training,  instructor’s  attitudes,  or  hardware  problems.  Faculty  should  learn  how  to 
use  technology,  but  not  completely  rely  on  the  technologies.  Instead,  they  should  be  able  to  identify  and  recognize 
the  strength  and  weakness  of  technologies,  and  select  the  most  appropriate  delivery  mechanism  for  their  lessons 
(Gunawardena,  1992,  as  cited  in  O’Quinn  & Corry,  2002). 

McGreal  and  Elliott  (2004)  summarized  the  technologies  being  used  in  today’s  online  instructional 
environment  as  multimedia,  streaming  audio,  streaming  video,  instant  messaging,  and  web  whiteboarding.  The 
authors  also  indicated  some  of  the  new  technologies  may  be  used  in  tomorrow’s  online  instruction  are  push 
technologies  and  data  channels,  audio  chat  and  voice  over  Internet  protocol,  hand-held  and  wireless  technologies, 
and  peer-to-peer  file  sharing. 
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New  interaction  and  communication  with  online  learners 

How  to  interact  with  online  learners  is  always  a challenge  for  instructors.  Muirhead  (2000)  reported  that  the 
teachers  he  interviewed  in  his  study  regarding  online  education  in  schools  noted  their  perplexity  on  how  to  initiate 
interactions  with  students  to  build  relationship  while  doing  their  online  teaching  job.  Because  most  faculty  are 
trained  in  “hand  to  hand”  teaching,  they  have  to  face  the  challenge  of  lack  of  direct  interpersonal  contact  with 
students,  and  they  have  little  contact  or  feedback  to  gauge  the  clarity  of  their  communications  (Bower,  2001).  Also, 
because  managing  electronic  course  materials,  student  participation,  student  achievement,  and  course  evaluations 
can  be  problematic  (Schott  et  ah,  2003),  interaction  with  students  appears  to  be  more  important  for  online  education 
instructors  on  encouraging  students  self-directed,  disciplined,  and  self-motivated.  Moreover,  because  of  the  needs  of 
different  interaction  methods  to  be  employed,  changes  are  also  needed  in  the  interpersonal  relations  between  the 
instructor  and  students  (Bower,  2001). 

New  way  of  learning  and  testing 

Since  the  face  to  face  instruction  is  usually  eliminated  in  online  classes,  instructors  may  lack  sufficient 
information  on  how  well  learners  actually  perform.  Thus,  ensuring  the  academic  honesty  and  integrity  is  another 
challenge  for  online  instructors.  Muirhead  (2000)  reported  that  all  online  teachers  in  his  study  worried  if  the 
completed  assignments  received  through  the  Internet  have  been  completed  by  students  themselves.  The  concerns 
expressed  by  those  teachers  Murihead  interviewed  may  also  partially  relate  to  other  online  educators’  complaints, 
relative  to  the  lack  of  direct  teacher  supervision  of  online  learning  and  testing.  McAlister,  Rivera,  & Hallam  (2001) 
raised  another  concern  about  the  difficulty  of  ascertaining  the  students’  identity  when  communicating  over  the 
Internet.  Cheating,  plagiarism,  and  integrity  in  taking  test  are  also  other  issues  in  ensuring  quality  online  instruction 
(Hanson,  2001;  Simonson  et  al,  2003).  While  many  critics  have  suggested  that  there  is  no  sure  way  to  hold  students 
accountable  for  academic  dishonesty,  Heberling  (2002)  concluded  that  while  maintaining  academic  integrity  in  the 
online  instructional  setting  may  be  a challenging,  he  asserts  that  many  strategies  may  be  employed  to  detect  and 
prevent  plagiarism,  such  as  reversing  an  Internet  search,  tracking  back  to  an  original  source. 

Berge,  Muilenburg,  & Haneghan  (2002)  grouped  identified  barriers  to  quality  online  instruction  into  10 
clusters.  Those  are  technical  expertise,  administrative  structure,  evaluation  and  effectiveness,  organization  change, 
social  interaction  and  quality,  student  support  services,  threatened  by  technology,  access  to  technology,  faculty 
compensation  and  time,  and  legal  issues.  Understanding  these  challenges  and  barriers  will  help  instructors  know 
how  their  roles  have  been  changed,  what  qualifications  they  need,  and  how  to  ensure  the  quality  of  online  education 
instruction  with  the  help  of  various  strategies. 

Facing  the  Challenge 

As  the  primary  key  to  ensuring  the  quality  of  online  education  instruction,  instructors  need  to  adjust  their 
attitudes  to  teach  online,  understand  what  qualifications  are  needed,  and  know  what  they  could  do  ensure  the  quality 
of  online  education  instruction.  As  Deubel  (2003)  has  argued  an  instructor’s  attitude,  motivation,  and  true 
commitment  affect  much  of  the  quality  of  online  education  instruction.  High  quality  online  education  instruction 
encourages  discovery,  integration,  application,  and  practices.  Instructors  need  to  discover  students’  learning 
preferences,  integrate  technology  tools,  apply  appropriate  instructional  techniques,  put  them  all  into  practices,  and 
generate  the  most  suitable  method  for  individuals.  Furthermore,  Cooper  (2000)  stated: 

Online  instruction  can  offer  new  challenges  and  opportunities  to  both  students  and  instructors.  Most 
students  do  not  view  online  instruction  as  a replacement  for  traditional  classroom  instruction.  However, 
with  the  right  subject  matter,  with  the  right  instructor  and  facilitator,  and  for  the  right  student,  Internet  or 
online  courses  can  provide  an  effective  educational  environment  that  is  a viable  alternative  to  traditional 
classroom  instruction,  (p.  54) 

Since  the  role  of  instructors  has  been  changed  in  online  education  courses  to  facilitator,  mentor  and  coach, 
the  instructors  will  need  to  adjust  their  attitudes  towards  technology  and  new  teaching  styles  to  meet  the  challenge. 
Attitudes  towards  technology,  teaching  styles,  and  control  of  technology  are  the  three  instructor  characteristics  that 
influence  learning  outcomes  (Webster  and  Hackley,  1997  as  cited  in  Volery,  2000).  Therefore,  students  are  likely  to 
experience  more  positive  learning  outcomes  when  their  instructors  hold  positive  attitudes  toward  online  delivery  of 
course  content  (Volery,  2000).  Contributing  factors  on  faculty’s  positive  attitudes  are  the  instructor’s  prior 
experience  of  teaching  online,  intellectual  change,  monetary  support  or  promotion/tenure,  availability  of  online 
courseware,  improved  training  and  facilities,  feedback  from  students,  and  flexibility  of  teaching  schedule  (Clay, 
1999).  To  become  an  online  teacher,  Deubel  (2003)  suggested  that  instructors  could  read  literature  about  online 
learning  environments  first,  and  then  get  trained  to  use  required  technology,  and  finally  seek  assistance  from 
experienced  instructors  when  needed. 
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In  order  to  design  and  deliver  effective  online  instruction,  instructors  should  know  what  qualifications  they 
must  have.  First,  they  need  to  upgrade  their  technical  skills  in  order  to  keep  abreast  of  technological  developments 
(Volery,  2000).  Second,  instructors  need  to  know  how  to  design  interactive  activities  and  course  syllabi,  how  to 
operate  the  learning  platform,  and  troubleshoot  with  problems  online  learners  may  encounter  (Cuellar,  2002). 
Therefore,  faculty  receiving  training  before  actually  delivering  online  education  courses  is  crucial. 

Many  researchers  have  reported  the  importance  of  faculty  training  (McKenzie,  Mims,  Bennett  & Waugh, 
2000;  Levy,  2003).  The  question  is  what  training  instructors  should  receive  to  qualify  them  to  deliver  online  courses. 
The  instructor  must  be  trained  in  using  the  designated  software,  managing  online  course,  integrating  web  sources, 
and  interacting  with  students  through  the  web  (Ko  & Rossen,  1998).  Some  online  facilitation  skills,  such  as  giving 
negative  feedback,  encouraging  students  to  become  actively  involve  in  online  learning,  and  dealing  with  disruptive 
students,  could  be  offered  in  training  programs  to  prepare  qualified  online  instructors  (Hitch  & Hirsch,  2001).  This 
training  is  best  offered  online,  since  it  provides  the  instructors  the  same  learning  experiences  as  their  students  (Ko  & 
Rossen,  1998;  Hitch  & Hirsch,  2001). 

Strategies  for  Designing  and  Delivering  Effective  Online  Instruction 

The  promise  for  effective  online  instruction  is  not  guaranteed  when  instructors  adjust  their  attitudes  to  new 
teaching  methods,  nor  when  they  receive  training  in  the  use  of  technology.  The  key  is  how  to  put  theory  into 
practice,  and  bring  them  both.  Instructors  should  understand  that  online  education  is  not  merely  uploading  teaching 
materials,  receiving  and  sending  e-mail  messages,  and  posting  discussion  topics  onto  the  Internet.  More  importantly, 
it  provides  an  arena  for  an  interactive,  deep,  collaborative,  and  multidimensional  thinking  and  learning  environment 
(Ascough,  2002). 

McAlister  et  al.  (2001)  suggested  that  a self-evaluation  process  in  the  online  courses  that  instructor’s  teach 
should  help  them  better  prepare,  design  and  deliver  online  courses.  The  self-evaluation  questions  might  contain: 
what  are  the  congruence  between  the  web-curriculum  and  the  institution’s  mission  and  strategy,  how  available  is  the 
administrative  support,  what  are  the  chances  of  institutional  obstacles,  what  are  issues  of  intellectual  property,  will 
any  compensation  from  institution  be  given,  how  to  select  the  courses,  how  available  is  assistance  of  facilities  and 
capabilities  on  preparation  and  delivery  of  the  course  material,  what  are  the  choices  of  instructional  methods,  how  to 
asses  student’s  progress,  how  to  adopt  a delivery  platform,  and  maintain  the  class  materials  online. 

Designing  an  Effective  Online  Learning  Environment 

To  ensure  the  quality  of  online  instruction,  the  online  learning  environment  must  be  designed  first  before 
the  instructor  embarks  on  the  online  course  delivery.  Wu  & Hiltz  (2004)  asserted  in  their  study  that  examined 
students’  learning  from  asynchronous  online  discussion  that  the  instructor  plays  an  important  role  in  motivating 
effective  online  discussion.  Therefore,  more  online  guidance,  more  structured  discussion  topics  and  considerate  time 
devotion  are  required  for  instructors. 

The  online  learning  environment  also  embraces  pedagogical  use  of  technology  (Ascough,  2002;  Yeung, 
2001),  integration  of  instructional  design  elements  (Zheng  & Smaldino,  2003),  various  types  of  medium  and  media 
(Deubel,  2003;  Palloff  & Pratt,  1999;  McAlister  et  al.,  2001),  and  diversified  learning  methods  include  deep 
learning,  critical  thinking,  collaborative  learning,  and  problem-based  learning  (Ronteltap  & Eurelings,  2002;  Rosie, 
2000;  Wheeler,  Waite  & Bromfield,  2002;  Ascough,  2002). 

Several  researchers  (Ascough,  2002;  Ronteltap  & Eurelings,  2002;  Rosie,  2000)  have  reported  that  online 
education  can  encourage  students’  deep  learning  and  critical  thinking  skills  when  learned  collaboratively  or  under 
problem-based  scenarios.  Ronteltap  and  Eureling’s  (2002)  experimental  study  revealed  that  when  students  are 
learning  in  a problem-based  practical  learning,  more  interaction  of  students  were  caused,  and  students  learn  more 
actively.  Therefore,  integrating  deep  learning,  critical  thinking,  collaborative  learning,  and  problem-based  learning 
methods  into  instruction  is  critical  to  instructors  in  improving  the  quality  of  online  education.  How  to  promote 
students’  deep  learning  via  online  education  is  a critical  factor  for  online  education  instructors  to  consider.  This 
requires  the  instructor  to  design  collaborative  and  problem-based  projects  which  will  involve  students  to  think 
critically,  actively,  and  deeply. 

To  ensure  the  effectiveness  of  the  online  learning  environment,  a detailed  course  plan  is  required.  The 
course  plan  should  include  doing  an  analysis  on  both  students’  and  instructor  needs,  class  objective,  selecting  course 
materials  for  students’  knowledge  construction,  designing  activities,  discussion  topics,  projects,  and  tests, 
envisioning  any  potential  problems  technically  or  academically,  and  testing  the  feasibility  of  the  online  course. 

In  developing  the  course  plan,  instructors  must  analyze  their  own  teaching  styles  first,  and  then  analyze 
learner’s  characteristics  (Ascough  2002).  Who  are  the  online  learners  and  how  fluently  can  they  use  computers  and 
the  Internet?  The  students’  learning  styles  should  also  be  examined.  Are  they  visual,  print,  aural,  interactive,  haptic, 
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kinesthetic,  or  olfactory  learners?  It  might  not  be  possible  to  gather  all  the  information  before  the  online  course 
begins,  but  a simple  online  survey  or  questionnaire  can  help  the  instructor  know  more  about  his  students’  learning 
styles.  One  type  of  questionnaire  could  be  a course  experience  questionnaire,  which  not  only  can  help  the  instructor 
to  gain  information  about  students,  but  can  also  improve  the  students’  perception  on  the  academic  quality  of  the 
course  (Richardson  & Price,  2003).  Paulsen  (1995,  as  cited  in  Palloff  & Pratt,  2003)  have  also  suggested  that 
incorporating  various  activities  can  successfully  address  all  learning  styles  of  the  virtual  student.  Those  activities 
could  be  one-alone,  one-to-one,  one-to-many,  and  many-to-many. 

When  organizing  the  content  for  online  education  courses,  the  learner’s  needs  must  be  taken  into  account. 
The  amount  students  learn,  their  ability  to  apply  learned  skills  into  practice,  and  their  satisfaction  with  the  learning 
experience  should  be  considered.  Evaluation  is  also  an  important  component  when  implementing  instructional 
design  principles  into  online  course  design,  because  it  is  the  way  to  gauge  students’  learning  outcome  and  the  quality 
of  course  instruction  (Zheng  & Smaldino,  2003). 

Instructors  should  keep  in  mind  that  online  learners  need  program  orientation  and  course  orientation  before 
getting  started.  The  program  orientation  should  be  offered  by  the  institution,  and  the  course  orientation  should  be 
provided  during  the  course,  as  well  as  by  the  institution.  Palloff  & Pratt  (2003)  recommended  that  the  program 
orientation  should  include  orientation  to  the  courseware,  basics  of  Internet  use,  how  and  where  to  get  help  when 
needed,  technology  requirements  for  online  courses  and  programs,  and  information  about  any  course  or  program 
policies.  The  authors  also  contended  that  course  orientation  should  provide  course  des  criptions,  syllabus,  faculty 
bios,  specific  information  on  course  expectations,  course  requirements,  assignments,  grades.  A “Frequent  Asked 
Questions”  file  about  the  course  and  how  to  complete  it,  as  well  as  course  or  program  policies  should  be  made 
available. 

Several  strategies  may  be  used  by  instructors  to  help  them  to  build  the  effective  learning  environment.  The 
strategies  include,  but  are  not  limited  to:  (a)  providing  background  information  for  the  course,  topics  on  the  unit,  key 
concepts  and  readings  for  the  course;  (b)  incorporating  PowerPoint  presentations,  video  lectures  and  demonstrations 
(this  is  especially  important  for  application  classes);  (c)  designing  some  activities  or  discussion  questions  which  can 
trigger  students’  interest  to  explore  the  answer,  which  will  ultimately  foster  students’  critical  thinking  and  deep 
learning;  (d)  requiring  students  to  play  roles  in  certain  scenarios  in  online  discussion  or  virtual  classroom.  Successful 
implementation  of  those  strategies  should  enormously  improve  the  quality  of  online  education  instruction. 

Allen  et  al.  (cited  in  Allen,  2001)  have  also  identified  10  keys  to  quality  online  learning.  The  authors 
suggested  that  online  courses  will  be  high  quality  when  they  are  student -centered  and  when: 

1.  Knowledge  is  constructed,  not  transmitted. 

2.  Students  can  take  full  responsibility  for  their  own  learning. 

3.  Students  are  motivated  to  want  to  learn. 

4.  The  course  provides  “mental  white  space”  for  reflection. 

5.  Learning  activities  appropriately  match  student  learning  styles. 

6.  Experiential,  active  learning  augments  the  Web  site  learning  environment, 

7.  Solitary  and  interpersonal  learning  activities  are  interspersed. 

8.  Inaccurate  prior  learning  is  identified  and  corrected. 

9.  “Spiral  learning”  provides  for  revisiting  and  expanding  prior  lessons, 

10.  The  master  teacher  is  able  to  guide  the  overall  learning  process,  (fl) 

Developing  an  Interactive  Online  Teaching -Learning  Community 

To  ensure  the  quality  of  online  education,  an  interactive  online  teaching-learning  community  should  be 
developed  by  the  instructor.  Unlike  instruction  in  the  traditional  classroom,  in  online  courses,  greater  attention  must 
be  paid  to  the  development  of  a sense  of  community  within  the  virtual  classroom  in  order  for  students’  learning  to  be 
successful  (Palloff  & Pratt,  2000).  This  online  community  will  augment  the  interaction  between  instructor-to- 
student,  student-to-student,  and  student -to-content.  Brown  (2001)  concluded  there  are  three  levels  of  community 
from  his  qualitative  study  on  a graduate  educational  administration  offered  by  a midwestern  university.  The  three 
levels  are:  (a)  making  on-line  acquaintances  or  friends;  (b)  building  community  conferment,  which  is  like  a 
membership  card  for  the  community  of  learners.  This  level  requires  online  learners  to  be  part  of  a long,  thoughtful, 
threaded  discussion  on  a subject,  (c)  camaraderie,  which  was  achieved  after  long-term  and/or  intense  association 
with  others  involving  personal  communication  (p.  24). 

According  to  Edelstein  & Edwards  (2002),  developing  an  effective  system  for  students’  ongoing  interaction 
is  one  of  the  chief  tenets  for  a successful  and  engaging  online  course.  The  characteristics  of  e-learning  community 
are  learner  centered,  active  learning,  instructor  guided  and  greater  participated  by  all  students  (Palloff  & Pratt, 

1999).  Ascough  (2002)  suggested  that  the  online  interaction  can  be  done  through  exploration,  reflection,  and 
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discussion,  which  ultimately  should  lead  to  students’  deeper  learning. 

The  e-mail,  listserv,  threaded  discussion,  and  chat  room  provide  an  efficient  communication  tool  to  build 
an  effective  online  community.  Threaded  discussions  could  be  a means  of  generating  or  promoting  interaction. 
Threaded  discussions  can  be  constructed  and  created  a home -like  atmosphere  by  instructor  whereby  students  can 
visit  and  embrace  the  joy  of  learning  (Edelstein  & Edwards,  2002).  In  this  environment,  the  interaction  between 
instructor-to-students  predominantly  consists  of  email  interactions  about  assignments,  questions  about  a particular 
aspect  of  a lesson,  and  general  messages  about  the  lesson.  The  student-to-student  interaction  is  mainly  discussing  the 
group  project,  or  discussion  questions  posted  by  the  instructor.  Setting  online  office  hours  may  be  a good  option  for 
the  instructor  to  bridge  the  gap  between  instructor  and  student  interaction,  since  students  can  get  immediate  answers 
to  questions  when  the  teacher  is  online  (Serwatka,  1999). 

Brown  (2002)  presented  several  tips  for  instructors  to  improve  the  impact  of  their  online  discussions, 
including:  (a)  maintaining  an  informal  tone  in  the  online  community  built  by  online  discussion,  (b)  relating  online 
discussions  to  issues  raised  and  happened  in  class,  (c)  structuring  discussion  topic,  stay  focused  around  a being 
solved  problem,  (d)  defining  roles  for  various  discussants,  such  as  “original  proposer”,  “idea  extender”, 

“constructive  critic”,  “responder  to  critic”,  or  “consolidator”,  (e)  providing  incentive  for  active  participant  in 
discussion  by  enhancing  grade,  (f)  requesting  backup  for  the  points  student  have  raised,  and  (g)  keeping  the 
discussion  board  to  be  a open  and  free  speech  platform  (p.  9.). 

Establishing  Performance  Assessments 

Reliable  and  valid  performance  assessments  should  be  established  by  instructors  for  quality  online  education 
instruction.  The  assessment  should  be  aligned  with  course  objectives  and  subject  aims,  and  should  enhance  students’ 
vocational  and  disciplinary  skills  (Morgan  & O’Reilly,  1999  as  cited  in  Zheng  & Smaldino,  2003).  An  assignment  is 
one  of  the  major  assessment  tools  used  to  measure  students’  performance.  To  ensure  the  quality  of  assignments  in 
the  online  learning  environment,  the  instructor  could  design  collaborative  assignments,  also  include  exemplary 
student  work,  permit  revision  of  students’  work,  and  encourage  students  to  initiate  course-related  discussion  topic 
(Deubel,  2003). 

Testing  is  another  assessment  tool  used  in  online  education  courses.  However,  due  to  the  special  features  of 
online  education,  teacher  and  students  might  not  meet  face  to  face.  Therefore,  academic  integrity  of  the  testing 
process  is  a crucial  issue.  One  way  to  ensure  quality  instruction  is  to  require  students  come  to  school  to  take  the  test, 
or  give  an  essay-type  test  alternatively  (Serwaktak,  1999).  Other  ways  to  prevent  the  cheating  in  tests,  according  to 
Olt  (2002),  would  be  to  disseminate  a special  username  and  password  to  students  prior  to  the  assessment  being  made 
available,  make  all  assessments  open -book,  set  time  limits  and  number  of  permissible  accesses,  randomized 
questions  from  question  pool,  and  use  courseware,  such  as  WebCT  to  track  the  time,  duration,  and  number  of 
attempts  that  a student  accesses  the  tests.  It  is  clear  that  the  quality  of  online  instruction  can  be  ensured  from  the 
instructors’  perceptive  when  they  hold  positive  attitudes  towards  teaching  online,  design  an  effective  learning 
environment,  develop  an  interactive  online  teaching-learning  community,  and  establish  reliable  and  valid 
performance  assessments. 

Assisting  Students  to  Achieve  Learning  Outcomes 

To  ensure  the  quality  of  online  education,  the  instructor  must  ensure  that  the  students’  learning  outcomes 
can  be  achieved.  However,  this  does  not  seem  as  easy  as  the  teaching  in  traditional  face-to-face  classroom.  Several 
researchers  have  expressed  their  concern  about  how  students’  learning  outcome  could  be  achieved  through  online 
education  (Wu  & Hiltz,  2004;  Koory,  2003).  Does  asynchronous  online  discussion  improve  students’  perceived 
learning 

(Wu  & Hiltz,  2004)?  Does  online  teaching  and  learning  have  particular  strength  in  ensuring  students’  learning 
outcome  (Simonson,  Smaldino,  Albright  & Zvacek,  2000)? 

As  early  as  in  1997,  Althaus  examined  142  undergraduate  students’  learning  outcomes  through  comparing 
the  blended  (face-to-face  and  computer-mediated)  discussion  and  the  traditional  classroom  discussion.  The  author 
found  that  this  combination  provides  a superior  learning  environment  compared  to  the  traditional  classroom  alone. 
Koorey  (2003)  taught  two  years  of  “An  introduction  to  Shakespeare”  at  the  University  of  California  Berkley.  One 
course  was  offered  online,  and  the  other  was  in  a traditional  face  to  face  (F2F)  class.  Through  two  years’  teaching 
and  observation,  the  author  found  that  her  online  students  achieved  dramatic  higher  learning  performance  than  her 
traditional  lecture  class.  The  author’s  conclusion  was  determined  by  whether  course  objectives  have  been  fulfilled, 
and  measures  of  course  grade  distribution.  Koorey  reported  that  fifty-eight  percent  (58%)  of  her  online  students 
received  an  A or  A",  as  opposed  to  the  more  usual  15%  of  students  making  similar  grades  in  the  face  to  face  course. 

Learning  outcomes  should  not  be  only  measured  through  students’  grades,  but  also  through  their  deep 
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learning,  higher  order  thinking,  critical  thinking,  or  problem-solving  skills.  Online  discussion  is  usually  regarded  as 
the  major  communication  tool  between  the  online  instructor  and  learners,  and  is  regarded  as  the  major  vehicle  to 
promote  deep  learning,  and  high  quality  learning  outcomes.  Larkin  -Hein  (2001,  as  cited  in  Wu  & Hiltz,  2004) 
reported  a research  study  addressing  the  role  of  students’  understanding  in  physics  using  an  online  discussion  group 
format.  The  author  found  that:  (a)  online  discussion  provided  an  additional  learning  and  teaching  vehicle,  (b ) online 
discussion  facilitated  the  acquisition  of  higher-order  thinking  skills,  and  (c)  students  became  more  adept  at 
transferring  and  applying  information  learned  in  class  to  novel  situations  (p.  141). 

In  addition  to  ensuring  the  online  students’  learning  outcomes  with  effective  online  teaching,  the  instructor 
should  be  able  to  accommodate  the  students’  learning  styles.  Koorey  (2003)  concluded  from  her  two  year  field  study 
that  students  who  possess  the  following  learning  styles  are  more  likely  to  be  successful  in  online  class:  experienced, 
self-directive,  task-oriented,  independent,  value  composed,  textual  communication,  less  social,  but  values  some 
give-and-take,  interested  in  problem-solving  and  immediate  application. 

Clark  (2002)  pointed  out  that  the  online  learner  must  be  a constructivist  learner.  This  suggests  that  the 
learner  must  be  active  in  the  process,  cognitively  complex  and  motivated.  According  to  Clark,  motivating  factors  in 
the  learning  process  include  self-reference,  personal  goals,  control  and  autonomy . Howland  & Moore’s  (2002) 
study  examined  48  students’  experiences  in  online  environments.  Their  results  confirmed  that  the  students  who  were 
the  most  positive  in  their  perceptions  of  online  learning  were  those  with  attributes  consistent  with  constructivist 
learners.  The  most  positive  students  were  more  independent,  proactive  and  responsible  for  their  learning. 

The  Role  of  Administrators  in  Ensuring  the  Quality  of  Online  Instruction 

The  administrator  should  be  a planner,  motivator,  promoter,  and  supporter  in  the  process  to  ensure  quality 
online  education.  To  ensure  the  quality  of  online  education  instruction,  administrators  should  plan  and  manage 
online  educational  programs,  and  support  faculty  balance  between  their  research  and  teaching  of  online  classes. 
Moreover,  Alley  (2001)  asserted  that  administrators  have  a distinctive  role  and  obligation  in  facilitating  quality 
learning.  He  encourages  administrators  to  examine  and  evaluate  online  education  programs  using  techniques  that 
are  aligned  with  quality  online  learning.  According  to  Alley  (2001)  this  review  will  ensure  that  faculty  design  web 
courses  for  construct  knowledge,  not  just  transmission  of  information;  develop  more  detailed  course  syllabuses  to 
include  timetables,  learning  tasks,  and  learning  outcomes;  plan  for  online  and  remote  assessment  sites  for  formative 
and  summative  assessments;  motivate  students;  accommodate  learning  and  teaching  styles  in  online  environments; 
and  promote  social  interaction.  Levy  (2003)  also  suggested  six  areas  to  consider  when  planning  online  program  in 
higher  education.  The  areas  are:  visions  and  plans;  curriculum,  staff  training  and  support;  student  services;  student 
training  and  support;  and  copyright  and  intellectual  property. 

The  administrator  should  motivate  faculty,  especially  senior  faculty  to  teach  online  courses  through 
intrinsic  or  personal  rewards,  such  as  tenure  and  promotion,  workload  adjustment,  or  reduction  in  duties  and 
increase  in  pay  (Giannoni  & Tesone,  2003).  Cuellar  (2002)  also  suggested  that  faculty  who  are  willing  to  teach 
online  should  be  provided  faculty  development  opportunities  to  order  for  them  to  learn  not  only  the  “technological 
know  how’s,  but  also  education  on  how  to  develop  courses  on  strategies  to  promote  interactive  online  learning’’  (p. 
11).  The  administrator  should  also  provide  and  arrange  administrative  and  technical  support  for  instructors  who 
teach  online  (Levy,  2003).  In  a study  conducted  by  Giannoni  and  Tesone  (2003)  to  determine  motivational  factors 
that  might  influence  participation  of  senior  faculty  in  online  learning  programs,  they  found  that  faculty  rated  release 
time,  personal  satisfaction,  e-teaching  development,  technical  support,  and  professional  prestige  as  motivational 
factors  that  influenced  their  participation  in  an  online  learning  program.  A consideration  of  these  factors  should 
assist  administrators  in  making  decisions  relative  to  the  involvement  of  faculty  in  online  education  programs. 

McKenzie  et  al.  (2000)  also  surveyed  faculty  needs  and  concerns  at  State  University  of  West  Georgia.  The 
authors  found  that  faculty  preferred  receiving  the  assistance  from  the  university  and  administrators  in  delivering 
online  courses  on  more  and  varied  training  sessions  (i.e.,  Authorware,  online  course  development),  technical  support 
as  needed  by  instructors  and  students,  more  time  to  design  and  deliver  on-line  class,  more  incentives  (i.e.,  laptop, 
student  assistants,  merit  pays),  helpful  support  services,  upgrading  WebCT,  limiting  the  online  class  enrollments, 
fixing  WebCT  problems  in  a timely  manner,  more  detailed,  understandable  instructional  material,  respecting  their 
wishes  to  teach  online,  making  WebCT  more  user  friendly,  and  mentoring  for  novice  or  less  experienced  online 
instructors,  and  more  time  to  adapt  using  WebCT  after  training). 

Husmann  and  Miller  (2001)  in  a study  of  the  perceptions  of  program  administrators  on  improving  distance 
learning  found  that  administrators  viewed  their  role  as  facilitators  of  program  quality.  The  findings  revealed  that 
administrator’s  perception  on  the  quality  of  an  online  program  is  based  almost  exclusively  on  the  performance  of 
faculty.  Therefore,  the  recruitment  of  qualified  faculty  to  teach  online  courses  becomes  extremely  important. 

Rahman  (2001)  suggested  a model  that  administrator  may  use  in  recruiting  faculty  to  teach  online  courses.  Within 
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the  model,  he  recommends  that  administrators  should  convince  the  perspective  candidate  about  the  principles, 
practices,  and  values  of  the  online  education,  and  the  online  programs  to  be  offered. 

Providing  supports  (training,  administrative,  monetary,  and  promotional)  and  hiring  qualified  instructor  is 
essential  for  administrators  to  ensure  the  quality  of  online  education  instruction  (McKenzie  et  al,  2000;  Husmann  & 
Miller,  2001;  Levy,  2003;  Giannoni  & Tesone,  2003).  Berge  (1998)  has  argued  that  online  teaching  and  learning 
will  definitely  fail  without  the  strong  support  of  administrators  on  programs,  trainings,  concerns  of  faculty  and 
students,  and  overcoming  barriers.  However,  Bowers  (2003)  contended  that  in  faculty  perspectives  must  be 
considered  in  order  for  quality  learning  to  occur  in  distance  education  technology  programs.  She  states, 
“administrators  must  "move  beyond  the  build  it  and  they  will  come  mentality”  (p.  4)  in  order  to  promote  greater 
faculty  participation  in  such  programs.  Therefore,  administrators  should  recognize  their  roles  in  educational 
institutions  and  determine  how  they  may  motivate  faculty  to  teach  online  courses  and  in  the  process  help  them  to 
ensure  the  quality  of  these  programs.  After  all,  a 2003  Sloan  Survey  of  Online  Learning  conducted  by  Allen  & 
Seaman  (2003)  revealed  that  academic  leaders  (59.6%  ) from  degree-granting  institutions  of  higher  education  agreed 
that  their  faculty  accepted  the  value  and  legitimacy  of  online  education,  while  over  40%  of  those  leaders  in  the 
institutions  surveyed  were  either  neutral  or  disagreed  that  faculty  embraced  online  education  as  a delivery  method. 
However,  the  findings  of  this  survey  also  showed  that  the  overall  attitudes  of  faculty  at  all  institutions  surveyed 
remained  more  conservative  with  regard  to  the  quality  of  online  education  and  its  ability  to  equal  face-to-face 
learning. 


Conclusion  and  Recommendations 

The  transition  from  the  traditional  face-to-face  classroom  to  on  line  learning  can  be  successfully  achieved 
and  quality  can  be  ensured  if  several  key  factors  closely  examined.  According  to  Palloff  and  Pratt  (2000),  those  key 
areas  are:  ensuring  the  access  to  and  familiarity  with  the  technology  used,  establishing  relatively  loose  and  free- 
flowing  guidelines  and  procedures,  striving  to  active  maximum  participation  of  participants,  promoting  collaborative 
learning,  and  enabling  online  participants  to  reflect  their  learning.  Levy  (2003)  also  suggested  that  when  planning, 
developing,  and  implementing  online  learning  programs  in  higher  education  six  factors  should  be  considered,  which 
are;  visions  and  plans,  curriculum,  staff  training  and  support,  student  training  and  support,  and  copyright  and 
intellectual  property  (p.  1).  Levy  concluded  that  if  institutions  want  to  have  effective  online  learning  programs  they 
must  analyze  all  of  these  areas  and  make  changes  as  necessary  in  order  to  successfully  implement  online  learning 
programs. 

What  is  considered  a good  online  course?  According  to  Keasley  (2000)  there  are  at  least  ten  critical 
elements  for  a good  quality  online  course;  They  are  content,  pedagogy,  motivation,  feedback, 
coordination/organization,  usability,  assistance,  assessment,  workload,  and  flexibility.  The  Institute  for  Higher 
Education  Policy  (IHEP  2000)  has  also  proposed  24  benchmarks  for  measuring  quality  Internet-based  learning 
which  should  be  considered  by  institutions  planning,  developing,  and  assessing  the  quality  of  their  online  learning 
programs. 

Some  studies  have  suggested  that  a team  approach  be  used  as  a method  to  ensure  the  quality  of  online 
education  instruction  (Care  & Scanlan,  2001;  Levy,  2003;  Southern  Regional  Education  Board,  2001).  Such  a team 
might  consist  of  the  instructional  designer,  graphic/interface  designer,  technical  support  personnel,  content  expert, 
direct  instructor,  information  resource  personnel,  mentors/tutors,  and  assessor.  The  instructor,  however,  remains  at 
the  center  of  the  team  to  guarantee  academic  integrity,  with  the  assistance  from  other  partners. 

To  ensure  the  quality  of  online  education  instruction,  the  qualification  of  instructors  should  be  the  first 
consideration  for  quality  assurance.  Instructors  who  conduct  online  education  courses  should  understand  what  their 
roles  are  and  adjust  their  attitudes  for  this  role  change.  Second,  it  is  important  for  instructors  to  master  design  and 
delivery  strategies,  techniques,  and  methods  for  teaching  online  courses.  Third,  the  institution  should  provide 
technical  and  financial  support  for  faculty.  Fourth,  school  administrators  should  also  realize  what  their  role  and 
responsibilities  are  in  ensuring  quality  online  instruction.  Critical  to  this  process,  administrators  should  recruit 
qualified  faculty  or  instructors  for  their  online  education  programs.  Moore  (2001)  also  noted  that  to  effectively 
deliver  online  courses,  faculty  must  promote  student  to  student  interaction  with  minimal  faculty  intervention, 
engage  students  in  regular  assignments,  promoting  students’  self-direct  ability,  and  providing  specialized  attention  to 
students  who  are  lack  of  self-directedness. 

The  increasing  diversity  of  the  nation’s  student  population  and  advancements  in  the  development  of 
educational  technology  has  encouraged  the  popularity  of  online  education  instruction  (Bi,  2000).  However, 
academic  institutions  that  offer  courses  online  still  face  many  challenges.  Therefore,  administrative  support  is 
crucial  if  programs  are  to  be  successful.  Administrators  must  consider  issues  related  to  intellectual  property, 
pedagogical  rigor  and  methods,  course  management,  and  instructional  compensation  of  faculty  (McAlister,  Rivera, 
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& Hallman,  2003).  In  essence,  successful  online  education  instruction  does  not  happen  by  magic.  It  is  a 
collaboration  of  instructors,  administrators,  students,  and  the  community  at  large.  The  courseware  development 
industries  should  also  keep  the  instructors  tuned  in  about  their  product  updates  and  provide  training  and  technical 
service  support  to  instructors.  The  government,  community,  and  parents  should  also  help  the  school  to  ensure  the 
quality  of  online  education. 

Moving  from  traditional  methods  of  teaching  to  online  methods  of  instruction  often  create  dramatic  shifts 
in  the  perspectives  of  instructors  and  their  students  (Dringus,  2000).  Moreover,  many  issues  have  been  raised  about 
the  quality  of  online  education.  To  resolve  some  of  the  problems  and  concerns  associated  with  online  education 
instruction,  Dringus  (2000)  suggested  that  administrators  and  faculty  should  prepare  students  for  the  online  learning 
experience.  Included  in  her  considerations  are: 

1.  Develop  a valid  and  reliable  pre -assessment  process  to  determine  the  educational  and  technical 
skills  background  of  the  learners  before  allowing  them  to  register  for  an  online  class. 

2.  Study  learners’  attitudes  and  perceptions  about  online  learning.  Determine  the  extent  of 
learners’  fears,  anxieties,  and  the  capacity  for  self-motivation  as  learners  begin  their  online 
courses. 

3.  Determine  the  extensive  stream  of  requirements,  roles,  and  responsibilities  that  must  be 
supported  and  maintained  in  online  classes  by  faculty,  students,  and  administrative  staff. 

4.  Demonstrate  to  potential  online  learners  how  unique  online  learning  is  compared  to  traditional 
campus-based  learning. 

5.  Devise  learning  or  study  strategies  that  students  can  adopt  to  maximize  their  online  learning 
experience,  (p.  194). 

Based  on  findings  in  this  review  of  the  literature  and  conclusions  in  this  paper,  the  following 
recommendations  are  made  for  ensuring  quality  online  education  instruction: 

1.  Administrators  should  not  force  faculty  to  teach  online  courses  who  do  not  wish 
to  do  so. 

2.  Training  in  WebCT  should  be  made  more  user  friendly. 

3.  Mentors  should  be  available  in  each  department  or  college  who  can  answer  questions  that  come  up 
from  faculty  who  have  limited  experience  in  teaching  online  courses. 

4.  Departments  should  limit  the  enrollment  in  online  courses. 

5.  Teachers  need  to  take  courses  to  better  understand  technology;  specific  classes  need 
to  be  taken  in  order  to  design  websites  for  online  courses. 

6.  Teachers  must  have  the  support  of  other  teachers  who  have  taught  online  courses 
before,  as  well  as  administrative  and  technical  support. 

Specific  recommendations  on  course  design  and  delivery  (Bi,  2000,  p.  42,  Bower,  2003) 

1.  Limit  the  size  of  an  online  class  because  timeliness  of  online  feedback  and  grading  suffers  when  there 
are  larger  numbers. 

2.  Determine  what  kind  of  learning  will  online  instruction  assist  and  how  that  will  shape  student 
learning? 

3.  Consider  carefully  the  type  of  instructional  design  to  be  used  to  increase  student  interaction  for  the 
objectives  of  the  course. 

4.  Determine  the  influence  of  multimedia  upon  the  instructional  process,  and  the  new  roles  of  the 
professor,  Web  developer,  site  facilitator,  and  non-traditional  students  in  the  distance  learning  process. 

5.  Provide  continuous  and  periodic  student  and  program  evaluation  to  assess  quality  indicators. 

6.  Faculty  should  be  encouraged  to  explore  the  online  environment  and  make  well  informed  decisions 
regarding  its  appropriateness  for  they  courses  that  they  teach. 

7.  Faculty  and  students  must  learn  how  to  use  the  online  tools  that  ensure  that  teaching  and  learning  is 
appropriate  for  academic  success. 
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Abstract 

How  to  ensure  the  quality  of  online  learning  in  institutions  of  higher  education  has  been  a growing  concern 
during  the  past  several  years.  While  several  studies  have  focused  on  the  perceptions  of  faculty  and  administrators, 
there  has  been  a paucity  of  research  conducted  on  students’  perceptions  toward  the  quality  of  online  education. 

This  study  utilized  qualitative  methods  to  investigate  the  perceptions  of  students  from  two  universities  and  one 
community  college  regarding  the  quality  of  online  education  based  on  their  own  online  learning  experiences. 
Interviews  and  observations  were  conducted  with  three  students.  Various  documents  were  collected,  digital  and 
printed.  Positive  and  negative  experiences  of  students  were  examined.  Factors  that  contribute  to  those  experiences 
were  also  identified.  The  findings  of  this  research  revealed  that  flexibility,  cost-effectiveness,  electronic  research 
availability,  ease  of  connection  to  the  Internet,  and  well-designed  class  interface  were  students’  positive 
experiences.  The  students’  negative  experiences  were  caused  by  delayed  feedback  from  instructors,  unavailable 
technical  support  from  instructors,  lack  of  self-regulation  and  self-motivation,  the  sense  of  isolation,  monotonous 
instructional  methods,  and  poorly-designed  course  content  The  findings  can  be  used  by  instructors  to  understand 
students’  perceptions  regarding  online  learning,  and  ultimately  improve  their  online  instructional  practices. 

Introduction 

With  the  fast  development  of  the  Internet,  many  colleges  and  universities  have  offered  online  courses  as  a 
viable  alternative  to  traditional  face-to-face  instruction.  However,  considerable  concerns  and  problems  have 
developed,  particularly  as  it  relates  to  the  quality  of  online  education.  Online  education,  according  to  Harasim 
(1989),  is  a new  domain  of  learning  that  combines  distance  education  with  the  practice  of  face-to-face  instruction 
utilizing  computer-mediated  communication  Ascough  (2002)  suggested  that  online  education  has  the  following 
features:  (a)  it  provides  a learning  experience  different  than  in  the  traditional  classroom  because  learners  are 
different,  (b)  the  communication  is  via  computer  and  World  Wide  Web,  (c)  participation  in  classroom  by  learners 
are  different,  (d)  the  social  dynamic  of  the  learning  environment  is  changed,  and  (e)  discrimination  and  prejudice  is 
minimized  (p.l). 

New  technologies,  the  Internet,  streaming  video,  net-meeting  etc.  now  makes  higher  education  more 
accessible  and  affordable  for  many  students,  and  for  those  who  would  have  been  unable  to  pursue  higher  education 
in  a traditional  in-class  setting  (Bianco  & Carr-Chellman,  2002).  Consequently,  online  learning  has  now  become  an 
integral  part  of  higher  education  institutions’  expanding  curriculum. 

The  term  online  education  is  often  associated  with  Internet  education,  virtual  education,  cyber-learning,  and 
asynchronous  learning  (Office  of  Sustainable  Development,  2000).  Kearsly  (2000)  reported  the  following  themes 
that  shape  online  education:  collaboration,  connectivity,  student-centeredness,  unboundedness,  community, 
exploration,  shared  knowledge,  multisensory  experience,  and  authenticity  (p.  4-10). 

Volery  (2000)  also  concluded  that  online  delivery  is  a form  of  distributed  learning  enabled  by  the  Internet. 

According  to  Paulsen  (2002),  online  education  is  characterized  by: 

• the  separation  of  teachers  and  learners  (which  distinguishes  it  from  face-to-face  education), 

• the  influence  of  an  educational  organization  (which  distinguishes  it  from  self-study  and 
private  tutoring), 

• the  use  of  a computer  network  to  present  or  distribute  some  educational  content 

• the  provision  of  two-way  communication  via  a computer  network  so  that  students  may  benefit 
from  communication  with  each  other,  teachers,  and  staff,  (p.l) 

Online  courses  and  degrees  have  been  widely  adopted  by  higher  education  institutions  as  another  method  to 
substitute  traditional  classroom  instruction.  Allen  and  Seaman’s  (2003)  recent  survey  on  online  education  delivered 
by  higher  education  institutions  in  the  United  States,  found  that  at  least  80%  of  the  course  content  delivered  by  those 
institutions  were  delivered  online.  Regardless  of  the  definition,  an  early  indication  of  the  widespread  popularity  of 
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online  education  courses  can  be  found  in  a survey  conducted  by  the  U.S.  Department  of  Education,  which  revealed 
that  more  than  54,000  online  education  courses  were  being  offered  in  1998,  with  over  1.6  million  student’s  enrolled 
(cited  in  Lewis,  et  al.,  1999).  In  a more  recent  study,  Allen  and  Seaman  (2003)  reported  that:  (a)  over  1.6  million 
students  took  at  least  one  online  course  during  the  Fall  of  2002,  (b)  over  one-third  of  these  students  (578,000)  took 
all  of  their  courses  online,  (c)  among  all  U.S.  higher  education  students  in  Fall  2002,  1 1 percent  took  at  least  one 
online  course,  and  (d)  among  those  students  at  institutions  where  online  courses  were  offered,  13  percent  took  at 
least  one  online  course  (p.l). 


Statement  of  the  problem 

Although  it  is  has  been  reported  in  a recent  study  that  80%  of  course  content  offered  in  institutions  of 
higher  learning  are  being  delivered  online  (Allen  & Seaman,  2003),  students  in  this  study  were  still  reluctant  to  take 
online  courses  and  complained  about  the  online  classes  they  had  taken.  One  participant  noted,  “Not  only  does  the 
courses  costs  more,  but  they  made  me  feel  lost  all  the  time”  (Personal  communication,  November  11,  2003). 

Another  participant  stated,  “The  online  class  was  very  boring,  and  I don’t  feel  the  instructor  helped  me  a 
lot’’(Personal  communication,  November  11,  2003).  It  appeared  that  these  students  held  unpleasant  experiences 
from  their  prior  online  learning  experiences.  What  caused  their  negative  experiences?  Was  it  the  learner  themselves? 
Was  it  the  program?  Or  was  it  because  of  the  instructor?  How  do  students  perceive  the  quality  of  online  education 
based  upon  their  own  online  learning  experiences?  Are  they  satisfied  or  dissatisfied  with  the  online  education  they 
have  received?  What  are  the  factors  that  shape  students’  online  learning  experiences?  All  of  those  questions 
prompted  the  present  study  and  its  investigation  to  explore  students’  perceptions  towards  the  quality  of  online 
education. 


Rationale  for  study 

As  the  number  of  online  education  courses  in  higher  education  has  increased,  concerns  and  issues  have 
arisen  about  the  quality  of  these  courses  (Yang  & Cornelious,  2003).  Many  problems  that  have  arisen  in  online 
education  regarding  its  quality  are  often  related,  but  not  limited  to:  (a)  the  requirement  of  separate  quality  assurance 
standards,  (b),  programs  having  low  (or  no)  quality  standards,  and  (c)  there  is  no  consensus  on  what  constitutes 
learning  quality  (Twigg,  2001). 

Camevale  (2000)  reported  that  Nick  Smith  (D,  Michigan),  the  chairman  of  the  House  of  Representatives 
science  subcommittee  on  basic  research  expressed  deep  concerns  about  the  quality  of  internet-based  courses  during 
a hearing  in  May  of  year  2000.  Representative  Smith  stated  that  he  remained  skeptical  of  the  quality  of  online 
learning,  “...  students  who  take  courses  online  don't  interact  as  much  as  their  peers  in  traditional  courses,  and  that 
they  may  walk  away  with  knowledge  but  not  with  an  understanding  of  how  to  think  for  themselves  (p.  5 1 .)” 
Concerns  have  also  arisen  as  to  the  use  of  technology  as  a panacea  to  correct  financial  problems  of  institutions  rather 
than  serve  as  a valid  teaching  method  (Hensrud,  2001).  Brown  & Green  (2003)  have  also  argued  that  online  course 
delivery  is  often  viewed  by  “administrators  as  a ‘cash  cow’  venue  - a means  of  delivering  instruction  to  a large 
number  of  paying  customers  without  the  expense  of  providing  things  such  as  temperature  controlled  classroom  and 
parking  spaces’’  (p.  148). 

Many  opponents  of  online  education  question  whether  or  not  online  learning  can  provide  the  same 
interaction  between  instructor-students  and  students  -students  as  traditional  classrooms  offered  (Roblyer  & Ekhaml, 

2000) .  Some  opponents  also  question  the  quality  of  online  education  since  the  quality  of  instructors  who  teach 
online  courses  cannot  be  guaranteed  (Weiger,  1998).  Arguments  are  made  that  as  consumers  of  online  education, 
students  are  unlikely  to  be  able  to  find  out  information  about  the  quality  of  the  courses  that  are  provided  (Twigg, 

2001) .  Schools  or  universities  that  offer  online  education  courses  typically  do  not  provide  comparative  information 
for  students  e.g.,  how  would  a student  know  which  online  course  meets  his/her  needs?  Moreover,  prerequisites  that 
are  essential  for  taking  a particular  online  course  are  usually  not  clearly  stated  on  websites  for  students,  and  when 
students  are  encountering  technical  problems,  who  will  they  be  able  to  ask  for  assistance  if  it  is  not  available  to  them 
(Twigg,  2001,  p.  15).  Thus,  additional  research  is  needed  to  examine  the  quality  of  online  education. 

Proponents  are  in  support  of  online  education.  They  have  suggested  that  the  lack  of  face-to-face  interaction 
can  be  substituted  by  online  discussions  in  bulletin  board  systems,  online  video  conferences  or  on  listservs  (Blake, 
2000).  Online  education  can  also  promote  students’  critical  thinking  skills,  deep  learning,  collaborative  learning,  and 
problem-solving  skills  (Ascough,  2002;  Rosie,  2000  & Briggs,  1999).  Donlevy  (2003)  asserted  that  online 
education  may  help  schools  expand  curricula  offerings  with  less  cost  and  can  help  graduates  gain  important 
technology  skills  to  improve  their  marketability.  Proponents  also  argue  that  online  education  can  encourage  non- 
discriminatory  teaching  and  learning  practices  since  the  teachers  and  students,  as  well  as  students  and  their 
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classmates  typically  do  not  meet  face-to-face.  Palloff  and  Pratt  (1999)  reported  that  because  students  cannot  tell  the 
race,  gender,  physical  characteristics  of  each  other  and  their  teachers,  online  education  presents  a bias -free  teaching 
and  learning  environment  for  instructors  and  students. 

Quality,  as  used  in  this  study  is  the  extent  to  which  an  internet-based  distance  education  program  meets 
the  benchmark  criteria  established  by  the  Institute  of  Higher  Education  Policy  in  2000  (IHEP  2000).  In  order  for  a 
distance  education  program  to  be  recognized  as  a quality  program  it  should  meet  these  specific  criteria  (Hensrud, 
2001).  According  to  Kearsley  (2000),  to  be  considered  as  good-quality  online  course,  ten  most  critical  elements 
must  be  incorporated.  They  are  “content,  pedagogy,  motivation,  feedback,  coordination/organization,  usability, 
assistance,  workload,  and  flexibility’'  (p.  105).  Numerous  research  projects  have  been  conducted  from  the 
perspectives  of  faculty  (Bennett  & Bennett,  2002;  Bower,  2001;  O’Quinn  & Corry,  2002;  Yueng,  2001)  and 
administrators  (Alley,  2001;  Giannoni  & Tesone,  2003;  Husman  & Miller,  2001)  toward  the  quality  of  distance 
education,  where  the  Internet  was  used  as  the  major  delivery  method,  based  upon  the  IHEP’s  quality  benchmarks. 
However,  there  is  a lack  of  research  to  measure  the  quality  of  online  education  from  the  students’  perspective.  Little 
is  known  about  the  quality  of  programs  that  offer  online  education,  especially  those  programs  based  on  the  Internet. 
Faculty,  administrators,  and  policy  makers  need  to  know  how  their  “customers”  view  the  quality  of  online  education 
programs  based  upon  their  own  learning  experiences. 

Purpose  of  the  Study 

Although  the  literature  regarding  online  education  is  expanding,  studies  related  to  the  quality  of  online 
education  are  limited.  Among  those  examined,  few  researchers  have  examined  the  quality  of  online  education  from 
the  students’  perspective.  Therefore,  there  is  a need  to  investigate  students’  perceptions  towards  the  quality  of  online 
education.  The  purpose  of  this  study  was  to  examine  the  quality  of  existing  online  education  courses  that  utilize  the 
Internet  as  the  primary  instructional  delivery  method.  The  focus  of  this  study  was  to  examine  students’  perceptions 
of  the  quality  of  online  education.  The  findings  of  this  study  may  contribute  to  the  literature  of  online  education  in 
terms  of  quality  assurance.  The  results  should  hopefully  enable  institutions  offering  online  education  to  evaluate 
their  programs  based  on  the  findings  and  the  recommendations  in  this  study. 

Research  Questions 

Answers  to  the  following  research  questions  were  sought  in  this  study. 

1.  What  is  the  experience  of  students  who  are  receiving  online  education?  How  do  they  perceive  the 
quality  of  online  education  from  their  experiences? 

2.  What  are  factors  that  have  shaped  students’  online  education  experience?  How  do  those  factors 
contribute  to  the  quality  of  online  education? 

Limitations 

There  are  several  limitations  of  this  study  which  need  to  be  addressed.  First,  the  three  students  who 
participated  in  this  study  were  taking  three  different  classes  offered  at  only  two  universities  and  one  community 
college.  Each  instructor  in  the  study  had  his/her  own  characteristic  in  regard  to  how  he/she  presented  course  content 
and  communicated  with  students.  Therefore,  the  characteristics  of  the  instructor  may  have  had  an  influence  on 
students’  perceptions  of  their  online  education. 

Second,  there  were  different  types  of  formats  utilized  to  present  the  online  courses.  Two  classes  used 
WebCT  as  courseware  technology,  and  one  class  used  Blackboard.  Although  there  are  many  similarities  for  the  two 
courseware  technology,  the  layout,  the  design  of  the  class,  and  interface  were  all  different. 

Third,  the  classes  were  across  disciplines  and  were  taught  at  different  levels.  One  was  a graduate  course  in 
Educational  Psychology.  One  was  an  undergraduate  course  in  Music  Appreciation,  and  the  third  was  a social 
development  class,  which  was  taught  at  the  undergraduate  level. 

Definition  of  Terms 

This  study  adopted  the  term  of  online  education  identified  by  Paulsen  (2002).  According  to  Paulsen, 
online  education  is  characterized  by 

• the  separation  of  teachers  and  learners  (which  distinguishes  it  from  face-to-face  education), 

• the  influence  of  an  educational  organization  (which  distinguishes  it  from  self-study  and 
private  tutoring), 

• the  use  of  a computer  network  to  present  or  distribute  some  educational  content 
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• the  provision  of  two-way  communication  via  a computer  network  so  that  students  may  benefit 
from  communication  with  each  other,  teachers,  and  staff,  (p.l.) 

Review  of  Literature 

Many  quantitative  studies  (Bennett  & Bennett,  2002;  Goodwin,  1993;  Hara  & Kling,  1999)  have  been 
conducted  in  an  effort  to  determine  the  effectiveness  of  on-line  learning.  However,  there  has  been  little  research  that 
has  sought  to  control  for  student  variables  that  could  provide  answers  to  the  following  questions  such  as:  How  do 
students’  computer  skills  affect  perceptions  of  on-line  quality?  Do  students’  computer  skills  also  affect  students’ 
learning  outcomes?  How  does  the  communication  within  the  on-line  environment  affect  student’s  perception  and 
learning  outcome?  According  to  Thurmond,  Wambach,  Connors  & Frey  (2002)  these  are  just  a few  of  the  questions 
that  are  often  ignored  or  under  investigated  in  research  that  has  assessed  the  quality  of  on-line  learning. 

Quality  assurance  guidelines  and  principles 

The  quality  of  online  education  has  also  prompted  the  attention  of  higher  education  accreditation 
associations.  Many  organizations  published  and  proposed  their  guidelines  or  principles  to  ensure  the  quality  of 
online  education.  In  the  early  1990s,  the  Western  Cooperative  for  Educational  Telecommunications  (WECT) 
developed  “Principles  of  Good  Practice  for  Electronically  Offered  Academic  Degree  and  Certificate  Programs” 
(Twigg,  2001).  Since  then,  many  other  groups  have  developed  similar  principles  and  practices.  For  example.  The 
American  Distance  Education  Consortium  (ADEC)  drafted  “ADEC  Guiding  Principles  for  Distance  Learning”.  A 
joint  task  force  of  the  American  Council  of  Education  and  the  Alliance:  An  Association  for  Alternative  Programs  for 
Adults  developed  “Guiding  Principles  for  Distance  Learning  in  a Learning  Society.”  The  Instructional 
Telecommunications  Council  provided  “Quality  Enhancing  Practices  in  Distance  Education.”  The  American 
Federation  of  Teachers  (AFT)  developed  “Distance  Education:  Guidelines  for  Good  Practice.”  The  Council  of 
Regional  Accrediting  Commissions  updated  and  explained  WECT’s  statement,  and  published  “Guidelines  for  the 
Evaluation  of  Electronically  Offered  Degree  and  Certificate  Programs”  (Twigg,  2001). 

In  2000,  The  Institute  for  Higher  Education  Policy  (IHEP)  first  reviewed  all  of  the  existing  principles  or 
guidelines,  and  proposed  24  benchmarks  for  measuring  quality  Internet-based  learning,  which  were  grouped  into 
seven  categories:  (a)  institutional  support,  (b)  course  development,  (c)  teaching/learning,  (d)  course  structure,  (e) 
student  support,  (f)  faculty  support,  and  (g)  evaluation  and  assessment  (IHEP,  2000).  Among  the  seven  categories, 
three  categories  are  related  to  students.  They  are  teaching/learning,  course  structure,  and  student  support.  The  IHEP 
student  Benchmark  scales  are  adopted  as  theoretic  framework  of  this  study  to  see  if  students’  perceived  good  quality 
of  online  education  is  congruent  with  IHEP  Benchmarks. 

Students’  perceived  strengths  of  online  learning 

Petrides  (2002)  conducted  a qualitative  study  to  determine  learners’  perspectives  on  web-based  learning. 
The  research  was  conducted  in  a blended  university  online  class,  which  means  the  class  was  a one-semester 
regularly  scheduled  class  with  web-based  technology  (LearningSpace)  as  a supplement.  When  interviewed,  some 
participants  indicated  that  they  tended  to  think  more  deeply  about  the  subject  areas  when  responding  in  writing  as 
compared  to  giving  verbal  responses.  They  explained  that  they  were  able  to  continually  reflect  upon  each  other’s 
reflections  because  of  the  public  and  permanent  display  of  the  discussion  postings  on  the  Web.  As  stated  by  one 
participant,  “There  is  something  that  forces  you  to  think  more  deeply  about  subject  areas  when  you  have  to  respond 
in  writing”  (Petrides,  2002,  p.  72).  Another  participant  reiterated  this  opinion,  indicating  that  the  online  technology 
allowed  more  reflection  than  in  face-to-face  classroom  discussion. 

Vonderwell  (2003)  interviewed  22  students  in  regards  to  their  perceptions  of  their  asynchronous  online 
learning  experiences.  Some  participants  expressed  that  the  asynchronous  environment  allowed  them  to  write 
carefully  about  their  ideas.  For  example,  Vonderwell  revealed  that  one  participant  stated,  “The  discussion  questions 
were  not  just  for  writing  the  answers;  they  required  reflection”  (p.  86). 

Flexibility  is  an  area  of  strength  of  the  online  learning  environment  that  has  been  identified  by  researchers 
(Petrides,  2002;  Schrum,  2002).  In  Petride’s  (2002)  study,  he  reported  that  participants  revealed  that  it  was  easier  to 
work  in  collaborative  groups  in  an  online  course,  since  there  was  no  less  needs  to  rearrange  everyone’s  schedule.  In 
addition  to  flexibility  with  time,  choices  related  to  the  learning  experience  were  also  reported  as  positive. 

Participants  in  Chizmar  and  Walber’s  (1999)  study  on  web-based  learning  environments  guided  by  principles  of 
good  teaching  practice  also  indicated  that  the  ability  to  freely  pick  and  choose  from  the  menu  of  diverse  learning 
experiences  enabled  them  to  find  the  approaches  that  best  fit  the  way  they  learn. 

Convenience  is  also  an  advantage  reported  in  the  online  learning  literature.  For  example,  in  Poole’s  (2000) 
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study  of  student  participation  in  a discussion-oriented  online  course,  the  findings  indicated  that  students  participated 
in  online  discussions  at  the  times  which  is  most  convenient  to  them,  such  as  on  weekends.  Poole  also  found  that 
students  mostly  accessed  the  online  course  from  their  home  computers,  which  was  the  place  most  convenient  to 
them.  Other  researchers  have  also  found  similar  results  that  online  learners  read  and  respond  to  instructor’s 
comments  in  online  discussions  at  times  convenient  to  them  e.g.  early  morning,  late  evening  (Murphy  & Collins, 
1997). 

Students’  perceived  weakness  of  online  learning 

Delay  communication  is  one  weakness  of  online  learning  that  is  reported  by  many  researchers  (Howland  & 
Moore,  2002;  Petride,  2002;  Hara  & Kling,  1999;  Vonderwell,  2003).  According  to  the  study  by  Howland  & Moore 
(2002),  the  communication  between  students  and  between  students  and  instructor  was  a critical  issue.  The  absence 
of  face-to-face  interaction  between  student  and  instructor  contributed  to  negative  perceptions  of  many  students. 
Students  felt  unconfident  in  guidance  when  the  feedback  from  instructor  was  delayed.  In  addition,  in  Howland  & 
Moore’s  study  (2002),  they  found  that  many  students  reported  that  it  was  difficult  to  get  clarification  on 
assignments,  etc.  due  to  lack  of  communication  between  student  and  instructor.  The  general  impression  of 
communication  between  students  was  also  negative.  The  message  board  was  the  main  communication  gateway 
between  students  and  instructor.  Each  student  was  required  to  make  a posting  on  message  board  each  week.  The 
students  often  reported  that  the  message  board  posting  was  ineffective  and  they  were  disappointed  in  the  level  and 
quality  of  communication  (Howland  & Moore,  2002). 

Petride’s  (2002)  study  on  learners’  perspectives  on  web-based  learning  also  reported  that  some  participants 
felt  a lack  of  immediacy  in  responses  in  the  online  context  in  comparison  to  what  could  typically  occur  in  a 
structured  face-to-face  class  discussion.  This  appears  to  be  especially  obvious  in  asynchronous  online  discussions, 
when  students  have  to  wait  for  others  to  read  and  respond  back  to  their  postings  or  e-mail  messages. 

Hara  and  Kling  (1999)  did  a qualitative  case  study  of  a web-based  distance  education  course  at  a major 
U.S.  university.  Their  participants  reported  the  lack  of  immediacy  in  getting  responses  back  from  the  instructor,  and 
as  a result  they  felt  frustrated.  Recent  studies  indicate  similar  results.  For  example,  in  Vonderwell’s  (2003)  study, 
one  reported  disadvantage  of  an  online  course  was  the  delay  of  immediate  feedback  from  the  instructor.  One 
participant  stated,  “It  might  take  hours,  maybe  a day  or  so  before  you  get  an  answer  back  for  the  question’’ 
(Vonderwell,  2003,  p.  84). 

Lack  of  a sense  of  online  community  and  the  feelings  of  isolation  were  other  weakness  that  learners  have 
reported  in  their  online  learning  experiences.  Vonderwell  (2003)  reported  that  online  learning  participants  indicated 
a lack  of  connection  with  the  instructor,  especially  “one-on-one’’  relationship  with  the  instructor.  Vonderwell 
revealed  that  one  participant  stated,  “I  still  feel  like  I know  a little  bit  about  my  instructor,  but  not  the  same  way  that 
I would  if  I was  in  a class.  I don’t  know  much  about  her  personality  at  all’’  (p.83).  Other  studies  have  found  similar 
results.  For  example,  Woods  (2002)  in  his  study  on  the  online  communication  between  instructor  and  learner 
reported  that  online  learners  reported  feeling  isolated  from  faculty  as  well  as  other  learners  in  the  online  courses  they 
had  taken. 

Factors  that  influenced  students’  online  learning  experiences 

There  are  many  factors  that  will  influence  students’  online  learning  experiences.  Song,  Singleton,  Hill  and 
Koh’s  (2004)  survey  study  on  76  graduate  students’  perceptions  of  useful  and  challenging  components  in  learning 
online  reported  that  lack  of  community,  difficulty  understanding  instructional  goals,  and  technical  problems  were 
challenges  in  their  online  learning  experiences.  Some  other  factors  identified  by  other  researchers  are  learner 
characteristics  (Howland  & Moore,  2002)  and  design  of  the  learning  environment  (Clark,  2002;  Dwyer,  2003;  Song 
et  ah,  2004). 

Learner  characteristics  that  influenced  students’  experiences 

Learner  characteristics  influence  the  way  online  learners  learn  and  their  online  learning  experiences. 
Howland  & Moore’s  (2002)  study  on  students’  perception  as  distance  learners  in  Internet-based  courses  revealed 
that  students  who  were  the  most  positive  in  their  perceptions  of  on-line  learning  were  those  with  attributes  consistent 
with  constructivist  learners.  The  most  positive  students  were  more  independent,  proactive  and  responsible  for  their 
learning.  In  contrast,  the  students  who  reported  more  negative  perceptions  of  their  on-line  learning  experience  had 
the  same  expectations  for  structure  and  information  as  they  did  for  an  in-class  format.  Those  students  with  negative 
perceptions  expressed  the  need  for  more  feedback  from  the  instructor  as  well  as  more  structure.  These  students 
reported  the  lack  of  feedback  and  communication  from  the  instructor  as  abandonment  (Howland  & Moore,  2002). 

Another  study  conducted  by  Garrison,  Cleveland-Innes,  and  Fung  (2004)  on  online  students’  role 
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adjustment  suggested  that  students  do  see  a difference  in  the  learning  process  and  a need  for  their  role  adjustment 
and  the  online  learning  should  be  viewed  as  more  cognitive  or  internally  oriented.  Garrison  et.  al  (2004)  also  pointed 
out  that  online  learners  must  take  more  responsibility,  adjust  to  a new  climate,  adjust  to  new  context,  synthesize 
ideas,  learn  how  to  participate,  synthesize  ideas,  apply  ideas  or  concepts,  and  stimulate  their  own  curiosity  to  be 
successful  in  online  class. 

Learning  Environment  that  influenced  students’  experiences 

Another  important  aspect  of  the  on-line  experience  is  the  design  of  the  on-line  environment  itself.  Clark 
(2002)  stated  in  Myths  in  E-learning  that  the  effectiveness  of  e-learning  “all  depends  on  the  quality  of  the  designed 
content”  (p.  599).  He  also  suggested  that  the  content  of  e-leaming  should  be  more  “meaningful,  distinct,  vivid, 
organized  and  personal”  (p.601)  to  increase  students’  retention. 

In  current  online  education  practices,  the  text  is  the  primary  means  of  communication.  Text  is  found  in 
traditional  paper  based  format,  multi-media  and  on-line  mediums.  These  texts  exist  not  in  isolation  but  within  a 
specific  context.  The  learning  style  and  learning  objectives  are  part  of  the  context  in  which  the  text  is  experienced. 
According  to  Dwyer  (2003)  the  use  of  text  alone  has  been  shown  to  be  unreliable  for  the  most  effective 
communication  between  individuals  lacking  shared  concrete  experiences.  However,  the  level  of  effectiveness  of 
textural  communication  is  enhanced  when  incorporated  with  feedback,  analogies,  questions  and  visuals. 

Images  can  convey  the  meaning  better  than  words.  As  Clark  (2002)  stated,  “A  picture  really  is  worth  a 
thousand  words  and  the  on-line  environment  can  take  advantage  of  the  ability  to  include  animation,  photographs, 
video  and  other  graphics”  (p.  601).  Dwyer’s  (2003)  meta -analysis  study  on  examining  the  effectiveness  of  text 
based  internet  learning  environment  revealed  that  the  inclusion  of  visual  images  in  the  learning  environment  can  be 
extremely  effective  but  the  inclusion  of  visual  images  should  be  based  on  specific  educational  objectives.  He  also 
found  that  visualizations  are  effective  according  to  the  type  of  visualization  and  the  type  of  learning  objective  and 
some  visual  aids  are  highly  effective  in  achieving  specific  learning  objectives. 

When  the  learning  environment  is  varied  rather  than  text  communication  only,  students  feel  more  satisfied 
with  their  learning.  Thurmond  et  al.’s  (2002)  study  of  evaluating  120  students’  satisfaction  in  a web-based  learning 
environment  asserted  that  the  virtual  learning  environment  including  emails,  computer  conferences,  chat  groups,  and 
online  discussions  has  a greater  impact  on  student  satisfaction  than  does  student  characteristics.  Song  et  al.’s  (2004) 
survey  study  also  asserted  that  design  of  the  course  is  one  of  the  helpful  components  in  their  online  learning.  Other 
helpful  components  included  comfort  with  online  technologies,  time  management  and  motivation  of  the  learner. 

Methodology 

Research  design  The  design  selected  for  this  research  study  was  qualitative  in  nature  using  interviews, 
observations,  and  documents.  Qualitative  research  provides  an  understanding  of  a situation  or  phenomenon  that  tells 
the  story  rather  than  determining  cause  and  effect  (Fraenkel  & Wallen,  2003;  Glesne,  1999). 

Techniques  for  conducting  qualitative  research  include  observations,  interview,  and  document  analysis. 
Triangulation — putting  together  various  types  and  pieces  of  information  -can  lead  to  a better  analysis  or 
interpretation  of  a situation.  According  to  Patton  (1990),  “Studies  that  use  only  one  method  are  more  vulnerable  to 
errors  linked  to  that  particular  method  than  studies  that  use  multiple  methods  in  which  different  types  of  data 
provide  cross-data  validity  checks”  (p.  18).  Interviews  and  observations  are  only  two  parts  of  that  process. 
Strengthening  the  information  collected  from  observations  and  interviews  with  other  data  is  not  essential,  but 
desirable  (Stake,  1995).  The  data  collection  techniques  used  in  this  study  consisted  of  the  structured  and 
unstructured  interviews,  observations,  and  documentations. 

Data  Collection  Data  were  collected  using  the  following  methods:  interviews,  observations,  and 
documents.  Multiple  sources  for  data  collection  will  allow  the  researchers  to  use  different  data  sources  to  validate 
and  crosscheck  findings  (Patton,  1990).  Documents  were  collected  and  evaluated  as  additional  resources  for  the 
study.  Each  participant  in  this  study  engaged  in  two  interviews.  One  of  the  interviews  was  structured  and  the  other 
was  unstructured.  Two  observations  were  conducted  in  an  effort  to  gain  more  qualitative  data.  The  observations 
were  approximately  60-minute  sessions.  The  observations  were  conducted  in  the  setting  that  participants  usually 
completed  their  online  coursework.  Documents  were  collected  from  all  participants.  These  documents  were 
printouts  and  other  information  to  substantiate  information  that  was  already  collected  and  witnessed  during  the 
observations  and  interviews  process.  Data  were  analyzed  to  answer  the  research  questions  and  to  cross  reference 
similarities  and  differences  among  participants. 
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Interviews  Fraenkl  & Wallen  (2003)  have  suggested  that  interviewing  is  an  essential  method  for  checking  the 
accuracy  of  the  impressions  that  a researchers  has  gained  through  observations  (p.  455).  Interviews  can  be 
conducted  in  several  ways-with  prepared  questions,  a protocol  or  interview  guide,  or  as  an  unplanned  event.  Patton 
(1990)  described  six  different  types  of  questions  that  all  contribute  valuable  information  to  the  research  study:  (a) 
experience/behavior  questions -what  the  individual  does  in  this  situation  or  has  previously  done;  (b)  opinion/value 
questions -what  the  individual  thinks  or  believes  about  a situation  or  issue;  (c)  feeling  questions -what  the  individual’s 
natural  emotional  responses  would  be  to  a situation  or  issue;  (d)  knowledge  questions -what  facts  the  individual 
knows  about  the  situation  or  issue;  (e)  sensory  questions -what  sensory  response  the  individual  would  have  to  the 
situation  or  issue;  and  (f)  background  or  demographic  questions-what  characteristics  describe  the  individual.  Each 
of  these  questions  guides  the  researcher  in  developing  an  understanding  of  the  phenomenon.  This  study  employed  a 
variety  of  questioning  techniques  (see  Appendix  A)  to  encourage  the  participants  to  describe  their  perceptions  of  the 
quality  of  online  education  based  on  their  own  experiences. 

Formal  structured  interviews  were  conducted  for  each  participant  at  the  beginning  of  the  study.  Each 
participant’s  interview  lasted  approximately  60  minutes  each.  The  interviews  were  conducted  at  the  convenience  of 
the  participants,  such  as  in  their  dorm  room,  office,  or  the  campus  library.  Unstructured  interviews  were  conducted 
with  each  participant.  The  unstructured  interviews  lasted  approximately  45-60  minutes.  Each  participant  had  an 
unstructured  and  a structured  interview.  Sample  questions  asked  during  the  formal  interviews  included:  As  a 
student,  how  does  your  experience  of  online  educational  programs  compare  with  traditional  in-class  instruction? 

How  do  you  like  or  dislike  it?  As  a student,  how  would  you  rate  the  overall  quality  of  the  online  instruction  you 
receive?  Very  good,  good,  moderate,  or  not  good?  Why?  In  what  ways  could  online  education  programs  serve 
your  educational  needs?  As  a student,  how  do  you  feel  about  the  communication  between  yourself  and  the 
instructor?  Between  you  and  other  students? 

Questions  for  the  unstructured  interviews  were  based  on  things  that  were  observed  or  were  not  seen  during 
observations.  Also,  much  room  was  allowed  for  spontaneous  questions  and  responses  from  both  the  participants  and 
the  researchers. 

Observations  Observations  were  conducted  from  February  to  May  during  the  spring  semester  of  2004.  An  integral 
part  of  qualitative  research,  observations  allow  the  researcher  to  determine  if  what  the  participant  has  said  in  the 
interview  is  transferred  into  action  during  the  online  learning  experience.  Participant  observation  “gives  a firsthand 
account  of  the  situation  under  study  and,  when  combined  with  interviewing  and  document  analysis,  allowed  for  a 
holistic  interpretation  of  the  phenomenon  being  investigated  (Merriam,  1998,  p.  102).  During  the  observation 
process,  field  notes  were  made  and  transcribed.  Pictures  about  the  setting  of  the  environment  where  the  participants 
normally  work  for  online  classes  were  also  taken. 

Participants  were  given  the  opportunity  to  schedule  their  observations  at  their  convenience  and  in  settings 
conductive  for  them.  Each  participant  was  observed  twice.  Some  of  the  observations  took  place  in  the  participant’s 
homes,  dorm  rooms,  offices,  or  the  campus  library.  During  the  process  of  observation,  extensive  field  notes  were 
taken.  Those  notes  included  participants’  study  habits,  description  of  the  physical  environment  where  the 
participants  were  studying,  and  how  the  participants  were  actively  or  inactively  involved  in  his/her  online  learning. 

Documentation  Fraenkel  and  Wallen  (2003)  indicated  that  documents  refers  to  any  kind  of  information  that  exists 
in  some  type  of  written  or  printed  form  intended  for  private  or  public  consumption  public  and  are  available  to  the 
researcher  for  analysis.  Thus,  photographs,  participants’  email  messages,  printed  version  of  the  class  layout,  and 
participants’  submitted  assignments  along  with  class  syllabus  were  collected  as  data  sources. 

Documents  concerning  online  education  were  collected  from  all  participants.  These  documents  included 
syllabus,  course  information,  tests,  study  notes,  discussion  board  postings,  emails , etc.  The  majority  of  the 
documents  received  from  the  participants  were  printouts  from  each  participant’s  course  site.  The  purpose  of 
implementing  observations,  interviews,  and  documents  was  to  provide  trustworthiness  and  accountability  to  the 
data.  The  researchers  were  attempting  to  determine  whether  the  participants’  actions  and  interview  responses  were 
the  same.  The  researchers  examined  how  closely  what  the  participants  said  in  the  interview,  and  observations  were 
aligned  with  what  actually  happened  in  the  typical  online  learning  settings. 

Participants  The  sample  for  this  research  study  were  three  students  enrolled  in  online  learning  courses.  The 
participants  received  online  instruction  from  two  different  universities  (University  of  Southern  Mississippi,  and 
Mississippi  State  University)  and  one  community  college  (East  Mississippi  Community  College)  in  Mississippi. 

Fraenkel  & Wallen  (2003)  have  concluded  that  purposive  sampling  is  based  on  the  assumption  that  one 
needs  to  “select  a sample  from  which  they  feel  will  yield  the  best  understanding  of  whatever  it  is  they  wish  to  study’’ 
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(p.  440).  However,  convenience  sampling  was  used  in  this  study.  Two  males  and  one  female  student  participated  in 
this  study.  Among  the  three,  one  was  Caucasian,  and  two  were  African  American.  These  participants  were  all 
enrolled  in  online  classes.  Their  exams  and  other  assessments  were  completed  online.  Two  of  the  participants 
received  instruction  delivery  via  WebCT  and  the  other  participant  received  instruction  via  Blackboard.  Two  of  these 
participants  were  non-traditional  students.  One  was  a housewife  of  approximately  42  years  old,  and  the  other 
participant  was  almost  40  years  old  and  held  a full  time  job.  Both  participants  took  undergraduate  courses  offered 
online.  The  third  participant  was  a traditional  student  of  25  years  of  age,  who  took  graduate  online  course. 

Procedures  The  initial  step  in  the  procedure  was  the  selection  of  study  participants.  Convenience  sampling  was 
done  because  the  participants  were  known  to  the  researchers.  Participants’  interviews  and  observations  were  then 
conducted  during  the  spring  semester  of  2004.  Structured  interviews  with  the  participants  were  first  conducted 
separately.  Conversations  were  recorded  during  the  entire  interview  processes.  Following,  the  participants  engaged 
in  unstructured  interviews  two  weeks  later.  The  interviews  were  conducted  in  different  places  to  accommodate  the 
needs  and  convenience  of  the  participants.  One  participant  was  interviewed  twice  in  his  dormitory  room.  The  second 
participant  was  interviewed  twice  in  her  home.  The  third  participant  was  interviewed  in  his  office  once  and  once  at 
the  campus  library. 

After  the  completion  of  interviews,  data  were  organized  and  preliminarily  analyzed.  The  first  observation 
was  then  scheduled  upon  the  participants’  consent  to  see  how  he/she  worked  for  the  online  class.  A week  later,  the 
second  observation  was  scheduled  with  participants  to  confirm  whether  or  not  there  had  been  any  change  in 
participants’  behaviors  when  the  participants  became  familiar  with  the  research  process. 

During  the  process  of  the  first  and  second  observations,  various  documents  were  collected.  Photographs 
were  taken  for  the  physical  setting  of  the  online  environment  of  the  participants.  The  online  class  design  and  layout 
were  also  printed  as  archival  data.  Some  other  documents  such  as  syllabus,  discussion  posting  messages  were  also 
collected  during  this  stage  of  the  process. 

Data  Analysis  Data  analysis  is  a key  component  of  qualitative  research.  Transcripts  made  of  audiotaped  interviews 
were  checked  for  accuracy  against  the  original  recordings.  Data  analysis  was  also  conducted  simultaneously  with 
data  collection.  Extensive  coding  of  the  interview  and  observation  transcripts  was  conducted  by  researchers.  The 
constant  comparative  method  (Glaser  & Strauss,  1967)  was  used  to  analyze  the  qualitative  data  from  different 
sources  over  time.  Data  were  organized  around  each  research  questions,  which  related  to  experiences  of  students 
who  were  taking  online  classes  and  the  factors  that  shaped  those  experiences.  The  researchers  examined  the 
interviews,  observations,  and  archival  data  for  similarities  and  differences.  This  information  was  then  complied 
around  two  major  areas.  These  areas  were  positive  experiences  and  negative  experiences  of  online  education. 

The  positive  experiences  included:  flexibility,  cost-effectiveness,  convenience,  self-paced,  availability  of 
technical  support,  and  ease  of  connection.  The  negative  experiences  included:  delayed  instmctor  feedback, 
unavailable  technical  support  from  instructor,  self-regulation  and  self-motivation,  and  a sense  of  isolation.  The 
factors  that  attributed  to  particpants’  positive  experiences  were:  easy  access  to  computers  and  Internet,  well- 
designed  course  layout,  available  technical  support  from  the  university  and  library,  spontaneous  grade  postings  after 
assessments,  and  flexible  class  participation  time.  The  factors  that  attributed  to  participants’  negative  experiences 
were:  untimely  or  lack  of  feedback  from  instructor;  monotonous  instructional  methods,  lack  of  technical  support, 
lack  of  interpersonal  communication,  and  poorly -designed  course  interface.  Limitations  of  the  analysis  is  reported 
and  discussed  in  the  findings  section  of  this  paper. 

Data  collection  and  analysis  provided  answers  to  the  following  research  questions:  (1)  What  is  the 
experience  of  students  who  are  receiving  online  education?  (2)  How  do  students  perceive  the  quality  of  online 
education  from  their  experiences?  (3)  What  factors  shaped  the  students’  online  education  experience?  (4)  How  do 
those  factors  contribute  to  the  quality  of  online  education? 

Findings  and  Discussion 

The  purpose  of  this  study  was  to  examine  students’  perceptions  towards  the  online  education  based  upon 
the  online  learning  experiences  they  had.  Students  perceptions  toward  online  education  based  on  their  own 
experiences  have  been  examined.  The  factors  which  shaped  those  students’  online  education  experiences  have  also 
been  investigated. 

The  findings  of  this  research  will  be  grouped  in  two  clusters:  students’  positive  experiences  and  negative 
experiences.  The  students’  positive  experiences  were:  flexibility,  cost-effectiveness,  electronic  research 
availability,  and  ease  of  connection  to  the  Internet.  The  students’  negative  experiences  were  identified  as:  delayed 
feedback  from  instructors,  unavailable  technical  support  from  instructor,  lack  of  self-regulation  and  self- 
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motivation,  and  the  sense  of  isolation.  Factors  that  contributed  to  students’  positive  experiences  were:  flexibility 
of  class  participation  time  and  self-paced  study,  cost-effectiveness  of  online  class,  electronic  research  availability, 
well-designed  course  layout,  ease  connection  of  the  Internet,  easy  navigation  of  the  online  class  interface,  and 
familiarity  with  the  instructor.  Factors  that  contributed  to  student’  negative  experiences  were:  delayed  feedback 
from  instructor;  unavailable  technical  support  from  instructor,  lack  of  self -regulation  and  self-motivation,  sense  of 
isolation,  monotonous  instructional  methods,  and  poorly -designed  course  content. 

Positive  experiences  and  contributed  factors 

1.  Flexibility 

The  flexibility  of  online  education  has  been  widely  recognized  as  one  advantage.  The  most  important  factor 
that  contributed  to  participants  ’ positive  experience  was  the  flexible  class  participation  time.  All  three  participants 
in  this  study  emphasized  the  convenience  they  enjoyed  from  not  being  required  to  drive  to  campus,  and  plan  their 
work  and  study  at  their  own  time.  Flexibility  with  time  was  one  positive  experience  found  in  this  study.  Students 
could  log  in  to  the  online  course  at  any  time  when  they  are  available.  “It  allows  me  to  take  to  log  on  at  will  and 
view  my  courses  and  assignments  at  will.”  “I  don’t  have  to  worry  about  trying  to  find  time  to  come  to  campus  or  a 
pointed  pace  to  meet  with  the  instructor.”  “One  benefit  of  online  course  is  that  you  don’t  have  to  worry  about  trying 
to  find  time  to  meet  as  a whole  class”.  “I  have  very  little  time  to  dedicate  to  coming  to  a campus  to  pursue  my 
education  because  I am  a full  time  mom.”  “There  is  no  hassle  in  trying  to  rush  to  get  to  class.”  “You’re  never  late 
for  class.”  One  of  the  non-traditional  students  reported  that  he  had  a full-time  job  that  was  extremely  demanding 
and  the  only  option  he  had  at  that  time  was  the  online  class.  This  student  had  strong  emotions  about  the  sacrifices 
that  he  perceived  his  family  had  made  due  to  his  pursuit  of  a higher  education  degree.  He  expressed  anger  towards 
the  educational  system  for  not  making  it  easier  to  attend  college.  “More  online  courses  would  have  been  a huge 
benefit... less  time  away  from  my  family.” 

Flexibility  with  self-paced  study  is  another  positive  experience  found  in  this  study.  The  participants  had 
full  control  of  when  to  study  the  required  knowledge  content  by  instructor.  It  was  considered  as  one  remarkable 
advantage  for  the  fulltime  employee  and  the  full  time  mother.  “You  can  pace  your  work  at  your  own  time  and  you 
don’t  have  to  listen  to  lectures  by  the  instructor.”  “Basically,  I get  to  work  around  my  own  time  schedule.”  “It  helps 
me  to  manage  my  time.”  “You  have  time  to  sit  back  and  reflect  your  initial  reactions  to  the  discussion  topics.” 

Participants  also  perceived  online  education  as  an  enjoyable  experience  when  studying  with  no  pressure 
from  the  instructor  and  the  other  students.  The  participants  felt  convenience  was  also  important  because  “There  is  no 
distraction  from  your  classmates”,  and  “There  is  no  one  looking  over  your  shoulder  or  checking  your  homework  or 
forcing  you  to  read.”  “You  can  pace  your  work  at  your  own  time  and  you  don’t  have  to  listen  to  lectures  by  the 
instructor.” 

2.  Cost-effectiveness 

All  the  three  participants  agreed  to  the  cost-effectiveness  of  online  education.  Although  they  have  to  pay 
extra  fora  fee  so-called  “Special  class  fee”,  compared  to  the  automobile,  gas,  textbook,  and  meal  cost,  they  would 
rather  take  online  classes  to  save  those  extra  expenses.  “I  think  online  courses  save  me  money.  I have  less 
automobile  cost  because  I don’t  drive  come  back  and  forth  to  campus.”  “I  spend  less  on  class  materials,  because  I 
fell  online  courses  require  fewer  books  if  any.  I don’t  spend  extra  money  on  backpacks,  notebook  papers,  parking 
decals  or  fines.  I feel  that  I make  more  efficient  use  of  my  money  while  taking  online  courses.” 

3.  Electronic  research  availability 

Electronic  research  availability  was  the  third  positive  experience  had  by  the  participants.  When  the 
graduate  student  was  required  to  do  some  research  work,  the  digital  library  was  his  first  choice.  The  student 
perceived  that  the  library  provided  a good  support  on  the  research  activities  in  the  online  class.  “As  far  as  research 
support,  our  library  did  a good  job.  We  can  use  online  databases.  If  we  have  any  questions,  we  can  call  the  librarian, 
the  librarian  can  help  us  to  answer  the  questions.”  “If  the  book  or  article  in  not  available  in  the  library,  we  can  use 
the  library  loan.”  “The  online  indexes  and  databases  are  open  to  all  students.” 

4.  Ease  of  connection  to  the  Internet 

The  ease  of  connection  to  the  Internet  is  the  fourth  positive  experience  found  in  this  study.  The  easy  access 
to  computer  and  Internet  stimulates  students’  interests  to  access  their  online  courses  quite  often.  Since  they  have 
access  at  home  or  at  the  dorm,  they  didn’t  need  to  drive  to  campus  or  school  to  access  their  online  class.  Some  could 
receive  discounts  on  Internet  access  due  to  the  enrollment  of  online  class.  “With  this  online  program  available 
through  internet,  getting  a degree  is  possible.”  “I  have  Internet  access  at  dorm,  I can  access  my  course  at  any  time, 
and  it’s  so  easy  for  me.”  “I  am  able  to  get  dial-up  Internet  access  cheap  from  university.” 

5.  Easy  navigation  of  the  online  class  interface 

The  well-designed  online  course  made  it  easy  for  students  to  navigate  and  find  the  information  they  were 


869 


seeking.  Two  kinds  of  courseware  technology  were  utilized  by  different  participant.  Two  participants’  online 
courses  utilized  WebCT,  and  one  participant  who  took  the  undergraduate  course  used  Blackboard.  Therefore,  there 
were  two  different  kinds  of  interfaces  being  used  in  this  study.  Two  students  stated  the  easy  navigation  for  the 
whole  class  design.  “The  menus  on  the  screen  on  the  Blackboard  are  very  easy  to  use.  I can  easily  navigate  my  way 
around  the  screen.”  “The  amount  of  links,  information  and  navigation  bar,  everything  is  very  simple  and  laid  out  on 
the  homepage.” 

Automatic  grading  for  tests  by  the  courseware  was  perceived  as  a good  experience  by  one  participant.  “I 
can  receive  the  results  for  the  tests  immediately  after  I took  it.”  A second  participant  also  reported  automatic 
grading  by  the  courseware  as  a positive  of  the  course.  Stating  that  it  was  great  to  see  not  only  your  grade  for  a 
specific  quiz  or  test,  but  to  have  a list  of  all  grades  received.  Another  participant  did  not  receive  automatic  grading 
and  reported  long  delays  in  reporting.  The  participant  stated,  “We’ve  already  taken  three  exams,  but  I still  haven’t 
gotten  the  result  from  my  first  test  yet.” 

6.  Familiarity  with  the  instructor 

One  interesting  phenomenon  found  in  this  study  was  the  comfort  level  or  familiarity  with  the  instructor. 

One  participant  indicated  that,  “I  feel  good  about  this  class,  because  I know  the  instructor  pretty  well.”  When  asked 
if  this  familiarity  brought  any  effect  on  his  perceptions  towards  the  quality  of  the  online  class,  the  participant 
answer,  “I  believe  this  familiarity  make  me  feel  more  comfortable  since  I already  knew  how  the  instructor  behaved 
and  his  way  of  teaching.” 

Negative  experiences  and  contributed  factors 

1.  Delayed  feedback  from  instructor 

The  delayed  feedback  from  the  instructor  was  conceived  as  the  main  factor  which  shaped  students’ 
negative  experience  on  the  quality  of  online  education.  Students  expect  to  receive  timely  feedback  from  instructors 
on  discussion  postings,  exam  or  tests,  and  submitted  assignments.  “The  feedback  from  the  instructor  is  not 
immediate.  So  far,  I haven’t  received  any  feedback  yet.  I think  there  is  a huge  need  for  improvements  as  regards  to 
the  feedback  from  the  instructor”.  “You  must  have  feedback  on  exams,  discussion  postings,  and  main  sources  of 
communication  on  the  regular  basis.”  Students  also  expect  the  instructor  to  reply  to  email  messages  or  voice  mail 
messages  promptly.  “She  (the  instructor)  didn’t  reply  to  my  calls,  or  my  email.  I had  no  way  to  contact  her.”  It 
normally  took  students  a day,  or  several  days  to  wait  for  feedback  from  the  instructor.  “I  have  to  wait  for  his 
responses,  it  normally  takes  a day  or  so.”  The  document  of  discussion  postings  of  one  class  showed  that  the 
instructor  only  replied  to  students’  messages  during  the  first  and  second  weeks  of  the  semester.  The  earliest  posted 
answer  by  students  was  Jan.  19th.  The  earliest  response  posted  by  the  instructor  was  Jan.  19th.  The  next  response 
posted  by  the  instructor  was  Jan.  29th.  By  that  this  time  the  total  amount  of  messages  that  had  been  posted  were  33, 
almost  one  third  of  Chapter  1 (89  messages).  Among  the  33  messages,  the  messages  posted  by  the  instructor  were 
ZERO.  No  wonder  the  student  complained  that  no  feedback  had  been  given  to  him. 

When  students  did  not  receive  feedback  from  the  instructor,  they  felt  frustrated,  depressed,  and  less 
motivated.  “This  miscommunication  killed  my  motivation  and  I almost  cancelled  the  class.”  “Sometimes  it  was  very 
easy  for  me  to  get  frustrated  when  I encounter  something  that  I don’t  understand.”  “He  is  probably  busy,  spending 
his  day  teaching  traditional  classes.  Therefore,  he  doesn’t  have  to  read  and  respond  to  emails  until  after  hours.” 
Participants  then  perceived  “Time  and  response  time  is  a drawback  in  all  aspects  of  online  education.”  During  the 
conversation,  the  interviewee  mentioned  several  times  about  the  frustration  he  experienced  because  of  delayed  or 
lack  of  feedback  from  the  instructor,  such  as  “I  hate  this  class,  no  feedback,  no  response.”  He  shook  his  head,  and 
sighed  deeply  when  giving  the  above  comment. 

2.  Unavailable  technical  support  from  instructor 

When  students  have  technical  problems,  they  need  someone  to  help  them.  The  person  that  came  to  their 
mind  first  was  the  instructor.  Thus,  when  the  technical  support  from  the  instructor  was  not  available,  negative 
experiences  will  be  brought  up.  One  participant  mentioned  that  she  and  her  classmates  had  no  technology 
background,  “Many  students  have  little  technical  background  knowledge  of  computers”.  It  was  even  difficult  for 
her  to  explain  the  problems  she  experienced  with  modern  technology.  “Some  problems  or  gray  areas  are  hard  to 
sum  up  in  words.  It  is  also  difficult  for  the  instructor  to  always  know  what  the  student  is  trying  to  say  or  describe.” 
Thus,  she  said  “I  have  to  seek  advice  from  friends,  who  were  computer  literate  about  how  to  send  and  compose 
messages.”  They  suggested  that  “A  how-to-mini  lesson  would  be  ideal.”,  or  “...  some  type  of  training  or  workshop 
on  WebCT  before  class  will  be  beneficial  to  us.” 

3.  Lack  of  self  regulation  and  self-motivation 

While  online  learners  enjoyed  the  flexibility  and  convenience  of  online  education,  they  also  needed  to  keep 
in  mind  that  they  had  to  take  some  responsibility  for  their  own  learning.  Due  to  the  freedom  and  convenient  nature 
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of  online  courses,  self-regulation  and  self-motivation  are  highly  expected  for  students  to  be  successful.  When 
students  can’t  control  him/herself,  he/she  might  miss  the  due  date  for  the  assignments,  or  even  the  dates  for  the  tests. 
One  participant  missed  the  deadline  for  one  of  his  tests.  He  had  to  keep  contacting  the  instructor  for  an  opportunity 
to  make  up  the  test.  However,  he  couldn’t  contact  the  instructor  through  any  means,  which  also  caused  his  negative 
experience  due  to  this  lack  of  connection.  The  participants  stated,  “You  must  be  really  self-motivated  and  focused”. 
“I  am  easily  distracted  and  I put  off  things  until  the  last  minute”.  Sometimes  they  even  complained  about  the 
freedom  and  independence  they  had  enjoyed,  “Independence  and  freedom  can  get  you  in  trouble.”  “It  is  so  easy  to 
not  to  complete  an  assignment.”  “You  don’t  have  a specific  time  to  do  your  work,  so  it’s  very  easy  to  get  off  track 
and  lose  focus.”  “If  I were  in  a traditional  class,  at  least  I can  talk  to  classmates,  and  we  may  remind  each  other 
about  the  due  dates  for  assignments  and  tests.” 

Participants  in  this  study  spent  their  time  on  housework,  babysitting,  other  coursework,  or  activities. 
Because  the  class  didn’t  meet,  sometimes  they  forgot  that  they  were  taking  online  courses.  “Sometimes  I will  forget 
my  assignments.”  However,  the  participant  realized  that  he  needed  to  control  himself,  and  motivate  himself,  “If  I 
could  have  spent  some  time  in  preparing  for  the  test,  I wouldn’t  have  gotten  such  a bad  score.”  He  also  realized  that 
he  spent  too  much  time  on  other  things,  such  as  talking  on  the  phone,  and  traveling.  “Sorry,  I talked  too  long  (he 
spent  30  or  more  minutes  talking  on  the  phone.”  “I  had  too  much  travel  this  year,  because  I had  to  go  for  an 
interview.” 

4.  Sense  of  isolation 

There  are  a variety  reasons  that  caused  a sense  of  isolation  for  online  learners  in  this  study.  One  reason  was 
the  lack  of  interpersonal  communication  or  interaction  between  instructor/student,  and  student/students.  “You  can’t 
talk  to  teacher  face  to  face  on  a regularly  basis  if  at  all.”  “You  miss  out  hands-on  experiences  done  in  class,  and  you 
won’t  be  able  to  interact  with  other  students.”  The  participant  felt  a strong  isolation  in  the  online  class  he  was  taking 
because  he  kept  saying  “I  feel  I am  nowhere  and  live  in  a lonely  island.”  Especially  when  there  is  no  group  work  for 
online  learners,  they  won’t  even  have  chance  to  talk  to  their  group  members.  Consequently,  students  don’t  know  the 
instructor,  and  don’t  know  his/her  classmates.  “So  far  I don’t  know  how  many  classmates  I have,  and  who  they 
are.”  “Although  the  instructor  listed  her  number  on  the  class  page,  but  I can  never  reach  her.” 

5.  Monotonous  instructional  methods 

Another  reason  that  caused  the  feeling  of  isolation  was  the  monotonous  instructional  methods  used  in  the 
online  class.  Two  participants  in  this  study  indicated  that  the  message  board  was  the  only  communication  and 
interaction  method  used  by  their  instructor.  “We  only  communicated  through  discussion  boarding.  She  (the 
instructor)  won’t  reply  to  our  email  message.  As  far  as  the  chatting  room,  we  never  used  it  in  this  class.  If  we  can 
meet  in  the  chatting  room,  it  will  be  so  much  better.”  The  monotonous  instructional  methods  also  included  class 
materials  the  instructor  prepared  for  students.  Printed  materials  were  solely  used  in  the  graduate  level  class.  In  the 
music  appreciation  undergraduate  level  class,  sound  files  and  graphics  are  also  utilized  as  supplementary  teaching 
materials.  Comparing  those  two,  the  participant  in  the  monotonous  learning  environment  expressed  his  feeling  and 
suggestion,  “Everything  is  printed.  If  we  could  have  different  material,  such  as  audio,  video,  or  even  let  us  rent  some 
video  and  write  a report  on  it,  I will  feel  better  for  my  learning.” 

6.  Poorly-designed  online  course  content 

A well-designed  course  interface  can  improve  students’  use  of  class  pages.  But  a poorly  designed  course 
interface  will  make  students  lose  in  seeking  information.  “I  don’t  know  where  she  (the  instructor)  put  the  page  for... 
(a  certain  assignment),  it  was  there  two  days,  but  now  it  is  gone.”  The  structure  of  one  course  design  was  not  logic. 
They  were  the  detailed  syllabus  and  the  extended  syllabus.  Under  the  extended  syllabus,  there  are  two  links,  one  is 
the  detailed  syllabus,  and  the  other  one  is  the  chapter  outlines.  But  the  chapter  outlines  were  not  linked.  The 
interviewer  also  noticed  the  inconsistent  design  of  the  course  from  the  archival  data  documents.  This  inconsistency 
caused  students’  confusion  and  frustration  in  finding  information  they  need. 

Participants’  overall  rating  of  the  quality  of  online  education  indicated  that  they  did  not  believe  that  they 
had  received  a good  quality  online  education.  In  addition,  from  their  experiences,  the  online  courses  they  had  taken 
did  not  meet  the  IHEP  benchmark  criteria.  IHEP  benchmarks  states: 

Feedback  to  student  assignments  and  questions  is  constructive  and  provided  in  a timely  manner.  Proper 
orientation  is  advised  to  students  before  starting  an  online  program.  Students  are  provided  with 
supplemental  course  information  about  course  objectives,  concepts,  etc.  Students  have  access  to  sufficient 
library  resources  that  may  include  “virtual  library”.  Students  are  provided  with  support  services,  such  as 
information  programs,  technical  and  proctoring  requirements,  training,  technical  assistance,  student  service 
personnel,  and  a structured  system  to  address  student  complaint,  (p.  2-3). 

One  positive  thing  indicated  by  all  participants  was  that  they  all  had  access  to  electronic  library  resources. 
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However,  they  all  expressed  their  dissatisfaction  on  the  delayed  feedback,  frustration  of  helplessness,  and  no 
orientation  before  the  online  classes  were  given.  Therefore,  the  participants  did  not  perceive  their  online  education 
to  be  of  high  quality. 


Conclusions  and  Recommendations 

The  purpose  of  this  study  was  to  gain  an  understanding  of  students’  perception  of  online  learning. 
Qualitative  research  methods  used  in  this  study  were  well  suited  to  achieve  this  goal.  The  depth  of  information 
gained  through  the  analysis  of  interviews,  observations  and  archival  data  have  provided  a level  of  understanding  that 
quantitative  methodology  could  not  have. 

While  receiving  online  education,  participants  gained  both  positive  and  negative  experiences,  although 
their  experiences  tended  to  be  more  positive.  Flexibility  of  class  participation  time  and  self-paced  study,  cost- 
effectiveness  of  online  class,  electronic  research  availability,  well-designed  course  layout,  ease  connection  of  the 
Internet,  easy  navigation  of  the  online  class  interface,  and  familiarity  with  the  instructor  contributed  to 
participants’  positive  experiences.  Factors  that  contributed  to  student’  negative  experiences  were:  delayed 
feedback  from  instructor;  unavailable  technical  support  from  instructor,  lack  of  self-regulation  and  self- 
motivation,  sense  of  isolation,  monotonous  instructional  methods,  and  poorly -designed  course  content. 

These  findings  were  supported  by  the  majority  of  the  research  literature  that  was  reviewed  in  this  study. 
Reflection,  flexibility,  and  convenience  are  reported  as  strengths  of  online  education  by  several  researchers 
(Petrides,  2002;  Vonderwell,  2003;  Poole,  2000;  & Murphy  and  Collins,  1997).  Some  contributing  factors  as 
regards  to  participants’  negative  experiences  could  also  be  found  in  similar  studies  found  in  the  literature.  As 
Howland  & Moore  (2002)  pointed  out  the  level  and  quality  of  communication  between  students  and  between 
students  and  instructor  was  a critical  issue.  Petride  (2002)  revealed  the  immediacy  in  responses  affected  learners’ 
experiences.  Vonderwell  (2003)  and  Song  et  al.  (2004)  also  reported  the  lack  of  a sense  of  community  in  students’ 
online  learning  experiences. 

This  study  also  found  the  feeling  of  familiarity  with  the  instructor  influenced  students’  learning 
experiences.  When  the  online  learner  knows  the  online  instructor,  he/she  may  feel  more  comfortable  while  taking 
the  instructor’s  online  class.  Based  on  this  finding,  a question  may  be  raised  as  to  whether  or  not  an  online  class 
should  be  taught  by  first-year  faculty?  Further  research  may  be  conducted  to  examine  the  effectiveness  of  online 
teaching  by  first-year  faculty  and  senior  faculty. 

When  participants  were  asked  to  evaluate  the  overall  quality  of  online  education  they  received,  their 
answers  were  moderate.  Moderate  quality  of  online  education  implies  that  they  were  not  very  satisfied  with  the 
education  received,  or  they  did  not  perceive  that  the  online  education  they  received  as  of  high  quality.  The 
participants’  personality  may  have  some  bearing  on  how  responsible  they  felt  for  their  own  learning.  Online  learner 
may  need  to  change  their  own  behaviors  such  as  lack  of  self-motivation,  spending  too  much  time  on  the  phone,  or 
not  being  an  active  and  constructive  learner.  However,  when  their  negative  experiences  were  examined,  all  of  the 
factors  except  the  one  related  to  learner  characteristics  (lack  of  self-regulation  or  self-motivation)  are  related  to  the 
online  instructor. 

Participant’s  in  this  study  felt  lost,  frustrated,  and  isolated  because  there  was  lack  of  immediate  response  or 
no  feedback  from  the  instructor.  When  the  course  content  was  not  organized,  it  increased  the  level  of  perplexity  and 
nervousness  of  online  learners.  When  participants  encountered  any  technical  problems,  the  instructors  were  not  able 
to  assist  them.  Even  when  the  instructor  could  not  help  them,  no  other  technical  person  could  help  the  online 
learners  either.  When  the  instructional  methods  were  only  through  textbooks,  or  discussion  postings,  students’ 
learning  was  not  effective.  The  learners  just  completed  assignment  and  turned  them  in  without  transferring  and 
assimilating  the  knowledge  from  textbook  to  their  own. 

The  authors  concluded  that  in  the  process  of  ensuring  the  quality  of  online  education,  the  instructor  plays  a 
key  role.  Not  only  because  the  instructor  “faces”  the  students  directly,  but  also  because  more  responsibility  has  been 
put  on  the  instructor’s  shoulder.  However,  this  does  not  mean  that  the  administrator  should  be  set  aside  in  the  quality 
assurance  of  online  education.  More  importantly,  the  administrator  should  provide  sufficient  supports  (training, 
administrative,  monetary,  and  promotional),  hire  qualified  faculty,  and  motivate  faculty  to  provide  effective  online 
teaching. 

This  study  was  conducted  with  students  from  two  universities  and  one  community  college  in  the  south. 
Convenience  sampling,  rather  than  another  sampling  technique  was  used.  If  a different  sampling  technique  had  been 
chosen,  the  results  could  be  different.  Therefore,  future  research  could  be  done  with  a homogeneous  group  of 
students,  using  a larger  sample  size;  including  more  universities  and  colleges  in  the  study. 

This  study  has  contributed  to  the  literature  in  the  area  of  on-line  education.  It  has  also  provided  valuable 
information  from  students  that  can  serve  both  online  instructors  and  administrators  in  providing  more  effective  on- 
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line  education.  This  study  suggests  that  IHEP  benchmarks  should  be  adopted  by  every  institution  of  higher 
education  as  a measure  of  the  online  programs  each  institution  offers.  When  the  online  education  programs  do  not 
meet  the  benchmarks,  students  won’t  perceive  their  online  education  to  be  of  high  quality.  This  study  has  also 
shown  that  more  research  need  to  be  conducted  in  the  area  of  improving  communications  and  utilizing  multi- media 
to  enhance  the  on-line  educational  experiences  of  students  in  regard  to  both  course  content  and  social  connectedness. 
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Appendix  A 
Interview  Questions 

1.  Asa  student,  how  does  your  experience  of  online  educational  programs  compare  with  traditional  in  - 
class  instruction?  How  do  you  like  or  dislike  it? 

2.  In  what  ways  could  online  education  programs  serve  your  educational  needs? 

3.  As  a student,  how  do  you  feel  about  the  communication  between  yourself  and  the  instructor? 
Between  you  and  other  students? 

4.  As  a student,  do  you  think  your  learning  outcomes  could  be  achieved  through  online  education? 
Why?  How? 

5.  As  a student,  how  do  you  view  the  feedback  from  instructor?  Is  it  in  a timely  manner?  Constructive? 
Please  give  some  examples. 

6.  As  a student,  how  do  you  think  the  technical  support  provided  from  university?  Do  you  receive  any 
other  type  of  support,  such  as  enrolling  in  online  class,  electronic  database,  and  written  information 
about  the  program?  If  you  have  any  complain,  is  there  anyone  you  can  address  to  and  solve  your 
problem? 

7.  How  do  you  view  your  online  educational  environment?  E.g.  Quality  of  graphics,  layout,  user 
friendly,  navigation,  etc.? 

8.  How  does  the  amount  of  course  work  in  your  online  education  program  compare  with  traditional  in  - 
class  instruction? 

9.  As  a student,  what  could  you  do  to  improve  the  quality  of  your  online  education? 

10.  What  do  you  think  are  the  important  factors  determining  the  quality  of  the  online  instruction  you 
receive? 

1 1 . What  factors  would  lead  you  to  choose  online  educational  programs  rather  than  traditional  in-class 
instruction? 

12.  As  a student,  how  would  you  rate  the  overall  quality  of  the  online  education  you  receive?  Very  good, 
good,  moderate,  not  good?  Why? 


874 


References 

Alley,  L.  R.  (2001).  What  makes  a good  online  course?  The  administrator’s  role  in  quality  assurance  of  online 
learning  [Electronic  version].  Converge,  4(11),  50,  52-53. 

American  Council  on  Education.  (November,  2003).  Guiding  principles  for  distance  learning  in  a learning  society. 
Retrieved  December  3,  2003,  from  http://www.acenet.edu/calec/dist_learning/dl_principlesIntro.cfm 

American  Distance  Education  Consortium.  (January,  2003).  ADEC  guiding  principles  for  distance  teaching 
and  learning.  Retrieved  December  3,  2003,  from 
http://www.adec.edu/admin/papers/distanceteaching_principles.html . 

American  Federation  of  Teachers,  Higher  Education  Program  and  Policy  Council.  (2000).  Distance 

education:  Guidelines  for  good  practice.  Washington,  D.C.:  American  Federation  of  Teachers. 

Retrieved  December  3,  2003,  from  http://www.aft.org/higher_ed/downloadable/distance.pdf 

Beard,  L.A.  & Harper,  C.  (2002).  Student  perceptions  of  online  versus  on  campus  instruction.  Education,  722(4), 
658-664. 

Bennett,  J.F.  & Bennett,  L.B.  (2002).  Assessing  the  quality  of  distance  education  programs:  The  faculty’s 
perspective.  Journal  of  Computing  in  Higher  Education,  13(2),  71-86. 

Biance  M.  B.  and  Carr-Chellman,  A.  A.  (2002).  Exploring  Qualitative  Methodologies  In  Online  Learning 
Envrionments.  The  Quarterly  Review  of  Distance  Education,  i(3),  251-260. 

Bower,  B.L.  (2001,  July  1).  Distance  education:  Facing  the  faculty  challenge.  Online  Journal  of  Distance 
Learning  Administration,  5(3).  Retrieved  October  4,  2003,  from 
http://www.westga.edu/~distance/ojdla/summer42/bower42.html 

Buck,  J.  (2001).  Assuring  quality  in  distance  education.  Higher  Education  in  Europe,  26(4),  599-602. 

Carnevale,  D.  (2000).  U.S.  lawmaker  questions  quality  of  the  online -learning  experience.  Chronicle  of  Higher 
Education,  46(38),  51. 

Chizmar,  J.F.  & Walber,  M.S.  (1999).  Web-based  learning  environments  guided  by  principles  of  good  teaching 
practice.  Journal  of  Economic  Education,  248-264. 

Clark.  D.  (2002).  Psychological  myths  in  e-learning.  Medical  Teacher,  24(6),  598-604. 

Cleary,  S.  (2001,  March  12).  The  downside:  Why  some  critics  give  Web-based  education  less-than-stellar  grades. 
Wcdl  Street  Journal . p.  R32. 

Driver,  M.  (2002).  Exploring  student  perceptions  of  group  interaction  and  class  satisfaction  in  the  web-enhanced 
classroom.  Internet  & Higher  Education, 5(1),  35-46.  Abstract  retrieved  December  3,  2003  from 
EBSCOHost  database. 

Dwyer,  F.  (2003).  Assessing  Strategies  for  Developing  Effective  and  Efficient  Text  for  Distance  Education: 
Traditional  and  Electronic,  International  Journal  of  Instructional  Media,  50(1),  11-23. 

Fraenkel,  J.  R.  & Wallen,  N.  E.  (2003).  How  to  design  and  evaluate  research  in  education  (5th  ed.).  New  York: 
McGraw-Hill. 

Garrison,  B.,  Cleveland-Innes,  M.  & Fung,  T.  (2004).  Student  role  adjustment  in  online  communities  of  inquiry: 
Model  and  instrument  validation  [Electronic  version].  Journal  of  Asynchronous  Learning  Network,  8(2), 
61-74. 

Giannoni,  K.L.  & Tesone,  D.V.  (2003,  March  20).  What  academic  administrators  should  know  to  attract  senior  level 
faculty  members  to  online  learning  environments?  Online  Journal  of  Distance  Learning  Administration, 
6(1).  Retrieved  September  25,  2003,  from 

http://www.  westga.edu/~distance/ojdla/spring6  l/giannoni6 1 .html 

Glaser,  B.G.  & Strauss,  A.L.  (1967).  The  discovery  of  grounded  theory:  Strategies  for  qualitative  research. 
Hawthorne,  NY:Aldine. 

Glesne,  C.  (1999).  Becoming  qualitative  researchers  (2nd  ed.).  New  York:  Addison  Wesley  Longman. 

Goodwin,  B.N.  (1993).  A study  of  the  perceptions  and  attitudes  exhibited  by  distance  education  students  and  faculty 
at  the  University  of  Phoenix  online  programs.  Unpublished  doctoral  dissertation,  Colorado  State 
University. (UMI  No.9424186). 

Hancock,  A.  (1999).  The  evolving  terrain  of  distance  learning.  Satellite  Communication,  23(3),  24-28. 

Hara,  N.,  & Kling,  R..  (1999).  Students’  frustration  with  a web-based  distance  education  course.  First  Monday, 
4(  12).  Retrieved  April  5,  2004,  from  http://www.firstmonday.com.dk/issues/issue4_12/index.html 

Hensrud,  F.C.  (2001).  Quality  measures  in  online  distance  education  at  a small  comprehensive  university. 
Unpublished  doctoral  dissertation.  University  of  Minnesota,  Duluth.  (UMI  No. 3010529). 

Howland,  J.L.  & Moore,  J.L.  (2002).  Student  perceptions  as  distance  learners  in  Internet-based  courses.  Distance 
Education,  23(2),  183-196.  Abstract  retrieved  November  18,  2003  from  EBSCOHost  Database. 

Husmann,  D.E.  & Miller,  M.T.(2001,  September  30).  Improving  distance  education:  Perceptions  of  program 


875 


administrators.  Online  Journal  of  Distance  Learning  Administration,  4(3).  Retrieved  October  4,  2003,  from 
http://www.westga.edu/~distance/ojdla/fall43/husmann43.html 
Institute  for  Higher  Education  Policy.  (2000).  Quality  on  the  line:  benchmarks  for  success  in  Internet-base  distance 
education.  Washington,  DC:  Institute  for  Higher  Education  Accreditation.  Retrieved  October  4,  2003,  from 
http  ://w  ww . ihep  .org/Pubs/PDF/Quality  .pdf 

Kearsley,  G.  (2000).  Online  education:  Learning  and  teaching  in  cyberspace.  Belmont,  CA:  Wadsworth. 

Lewis,  L.,  Snow,  K.,  Farris,  E.  & Levin,  D.  (1999,  December  17).  Distance  Education  at  Postsecondary 

Education  Institutions:  1997-98.  National  center  for  educational  statistics.  Retrieved  on  November 
18,  2003  from  http://nces.ed.gov/pubs2000/2000013.pdf 

Merriam,  S.B.  (1998).  Qualitative  research  and  case  study  applications  in  education.  San  Francisco,  CA:  Jossey- 
Bass. 

McGregor,  E.N,  & Attinasi,  L.C.  (1998).  The  craft  of  benchmarking:  finding  and  utilizing  district-level,  campus- 
level,  and  program-level  standards.  Paper  presented  at  the  Rocky  Mountain  Association  for  Institutional 
Research  Annual  Meeting,  October,  Bozeman,  MT.  (ERIC  Document  Reproduction  Service  No. 
ED423014). 

Moursund,  D.  (1997).  The  future  on  information  technology  in  education.  Learning  and  Leading  with  technology, 
25(1), 5. 

Murphy,  K.L.  & Collins,  M.P.  (1997).  Communication  conventions  in  instructional  electronic  chats.  First  Monday, 

2 (11).  Retrieved  April  5,  2004,  from  http://www.firstmonday.com.dk/issues/issue2_ll/index.html 
National  Governors’  Association.  (2001).  State  of  e-learning  in  the  States.  Washington,  DC:  Author. 

Retrieved  November  28,  2003,  from  http://www.nga.org/cda/files/060601ELEARNING.pdf 
Office  of  Sustainable  Development.  (2000,  November).  Connected  education.  Retrieved  December  4,  2003,  from 
Washington  State  University,  Knowledge  Exchange  and  Learning  Partnership  Networks  Web  site: 
http://cbdd.wsu.edu/networks/KelpWebSite/connected/ce-definition.html 

O’Quinn,  L.  & Corry  M.  (2002,  December  16).  Factors  that  deter  faculty  from  participating  in  distance 

education.  Online  Journal  of  Distance  Learning  Administration,  5(4).  Retrieved  October  4,  2003,  from 
http://www.westga.edu/~distance/ojdla/winter54/Quinn54.html 
Patton,  M.  Q.  (1990).  Qualitative  evaluation  and  research  methods(2nd  ed.).  Newbury  Park,  CA:  Sage 
Publications. 

Petrides,  L.  A.  (2002).  Web-based  technologies  for  distributed  (or  distance)  learning:  Creating  learner-centered 

educational  experiences  in  the  higher  education  classroom.  International  Journal  of  Instructional  Media, 
29(1),  69-77. 

Poole,  D.M.  (2000).  Student  participation  in  a discussion-oriented  online  course:  A case  study.  Journal  of  research 
on  Computing  in  Education,  33(2),  162-177. 

Phipps,  R.,  & Merisotis,  J.  (1999).  What’s  the  difference?  A review  of  contemporary  research  on  the 
effectiveness  of  distance  learning  in  higher  education.  A report  from  the  Institute  for  Higher 
Education  Policy.  Retrieved  December  1,  2003  from  http://www.ihep.com/Pubs/PDF/Difference.pdf 
Roblyer,  M.D.  & Ekhaml,  L.  (2000).  How  interactive  are  YOUR  distance  courses?  A rubric  for  assessing 
interaction  in  distance  learning.  Online  Journal  of  Distance  Learning  Administration,  3(3). 

Retrieved  November  15,  2003,  from  http://www.westga.edu/~distance/roblyer32.html 
Poole,  D.M.  (2002).  Student  participation  in  a discussion-oriented  online  course:  A case  study.  Journal  of 
Research  on  Computing  in  Education,  33(2),  162-177. 

Schrum,  L.  (2002).  Oh,  what  wonders  you  will  see:  Distance  education  past,  present,  and  future.  Learning  and 
Leading  with  Technology,  30(3),  6-9,  20-21. 

Song,  L.,  Singleton,  E.S.,  Hill,  J.R.  and  Koh,  M.H.(2004).  Improving  online  learning:  student  perceptions  of  useful 
and  challenging  characteristics  [Electronic  version].  The  Internet  and  Higher  Education,  7,  59-70. 

Stake,  R.  E.  (1995).  The  art  of  case  study  research.  Thousand  Oaks,  CA:  Sage  Publications. 

Twigg,  C.  (2001).  Quality  assurance  for  whom?  Providers  and  consumers  in  today’s  distributed  learning 
environment.  The  Pew  Learning  and  Technology  Program,  Center  for  Academic  Transformation, 

Troy,  New  York.  Retrieved  February  12,  2004  from  http://www.center.rpi.edu. 

Thurmond,  V.  A.,  Wambach,  K.,  Connors,  H.  R.  & Frey,  B.  B.  (2002).  Evaluation  of  Student  Satisfaction: 

Determining  the  Impact  of  a Web-Based  Environment  by  Controlling  for  Student  Characteristics.  The 
American  Journal  of  Distance  Education  , 16(3),  169-189. 

Vonderwell,  S.  (2003).  An  examination  of  asynchronous  communication  experiences  and  perspectives  of  students 
in  an  online  course:  A case  study.  Internet  and  Higher  Education,  6(1),  77-90. 

Waits,  T.,  & Lewis,  L.  (2003,  July  18).  Distance  Education  at  Degree -Granting  Postsecondary  Institutions:  2000- 


876 


2001.  National  center  for  educational  statistics.  Retrieved  on  November  18,  2003  from 
http://nces.ed.gov/pubs2003/2003017.pdf 

Weiger,  P.R.  (1998).  What  a tangle  (world  widejweb  we  weave.  Community  College  Week,  10(22),  1 1-13. 
Retrieved  October  4,  2003,  from  EBSCOHost  database. 

Wheeler,  S.  (2002).  Student  perceptions  of  learning  support  in  distance  education.  Quarterly  Review  of  distance 
education,  3(4),  419-430.  Retrieved  December  6,  2003  from  EBSCOHost  database. 

Woods,  R.H.  (2002).  How  much  communication  is  enough  in  online  course?  Exploring  the  relationship  between 
frequency  of  instructor-initiated  personal  email  and  learners’  perceptions  of  and  participation  in  online 
learning.  International  Journal  of  Instructional  Media,  29(4),  377-394. 

Yang,  Y.  & Cornelious,  L.  (2003,  November).  Ensuring  the  quality  of  online  education  instruction.  Paper 
presented  at  Mid  South  Educational  Research  Association  Conference,  Biloxi,  MS. 

Yeung,  D.  (2001,  December  20).  Toward  an  effective  quality  assurance  model  of  web-based  learning:  The 

perspective  of  academic  staff.  Online  Journal  of  Distance  Learning  Administration,  4(4).  Retrieved 
October  2,  2003,  from  http://www.westga.edu/~distance/ojdla/fall44/yeung44.html. 


877 


The  Value  of  Modularity  in  Instructional  Design:  Implications  for  Improved 
Validity  in  the  Evaluation  of  New  Techniques  in  Distance  Learning 

Kenneth  A.  Yates 
David  F.  Feldon 

University  of  Southern  California 

In  the  development  of  distance  learning,  advances  in  cognitive  science  merge  with  new  technology  to 
deliver  instruction  worldwide.  However,  one  major  difficulty  in  evaluating  the  efficacy  of  these  tools  is  determining 
which  elements  of  instruction  truly  lead  to  observed  changes  in  student  performance.  As  content,  pedagogical 
methods,  and  media  are  intertwined,  identifying  the  “active  ingredients”  is  an  essential  element  of  facilitating 
training  that  is  of  high  quality  and  minimizes  development  costs  (Clark  & Estes,  2000).  To  accurately  evaluate 
applications  in  the  field,  researchers  must  be  able  to  identify  specific  instructional  components,  make  decisions  on 
who  and  what  will  be  subject  to  treatment,  and  accurately  draw  inferences  regarding  causal  interactions  without  the 
control  offered  by  a laboratory.  The  purpose  of  this  paper  is  to  briefly  review  current  use  of  various  research 
methods  for  evaluating  instructional  technologies,  discuss  previous  solutions  to  balancing  the  conflicting  demands 
of  internal  and  external  validity,  and  then  to  propose  a new  research  design  that  achieves  this  goal  in  a manner 
compatible  with  many  instructional  technology  applications. 

Current  Methodologies  in  Instructional  Technology  Research 

As  educational  researchers  have  grappled  with  the  challenges  of  obtaining  robust  findings  that  successfully 
transfer  from  the  laboratory  to  the  field,  use  of  experimental  and  quasi-experimental  designs  has  decreased  in  favor 
of  contextually  embedded  case  studies  that  are  descriptive  in  nature  (Winn,  2002).  Emphasizing  the  post  hoc 
measurement  of  past  learning  outcomes  or  present  conditions,  it  has  been  asserted  that  descriptive  research  may 
often  be  the  only  feasible  way  to  study  relationships  between  educationally  relevant  variables.  Describing  and 
interpreting  “what  is”  makes  descriptive  research  particularly  appropriate  to  study  questions,  where  significant 
variables  cannot  be  manipulated  in  an  authentic  setting  without  being  detrimental  or  threatening  to  human  subjects 
(Best  & Kahn,  2003). 

The  movement  toward  qualitative  studies  can  be  examined  from  the  perspective  of  new  entrants  to  research 
in  instructional  technology.  Caffarella  (2000),  analyzed  the  content  of  doctoral  dissertations  in  educational 
technology  from  1977  through  1998  and  found  a clear  shift  in  the  selection  of  research  methodologies.  Over  that 
time  span,  he  observed  a reduction  in  the  number  of  experimental  studies  and  an  increase  in  the  number  of 
qualitative  studies.  Noting  that  his  selection  method  most  likely  underestimated  the  actual  percentages  and  that 
doctoral  studies  tend  to  be  concentrated  in  a few  institutions  and  professors,  he  concluded  that  “the  balance  between 
experimental  and  qualitative  studies  will  most  likely  continue  to  show  a change”  (p.  20). 

In  their  review  of  distance  education  research  from  1990  to  1999,  Berge  and  Mrozowski  (2001)  evaluated 
890  research  articles  and  dissertations  and  concluded  that  approximately  75%  involved  descriptive  research,  while 
only  6%  employed  a valid  experimental  approach.  Further,  their  analysis  of  the  data  indicated  that  pedagogical 
methods  dominated  the  research  and  that  the  trend  continues  to  increase.  Similarly,  Ross  and  Morrison’s  (2003) 
analysis  of  424  articles  published  by  Educational  Technology  Research  & Development,  and  its  predecessor 
publications  from  1953  through  2001,  found  a decline  in  the  use  of  true-experimental  studies  from  their  height 
(77%)  during  the  period  1983-1992,  to  53%  during  1993-2001,  while  descriptive  studies  increased  from  13%  to 
45%  during  the  same  time  periods.  They  suggested  that  this  demonstrates  the  growing  influence  of  alternative 
designs,  such  as  case  studies. 

Their  analysis  of  the  types  of  stimuli  and  assessment  used  in  these  studies  was  also  noteworthy,  because  it 
revealed  a significant  decrease  in  the  use  of  materials  that  were  developed  only  for  the  purposes  of  conducting  the 
reported  study,  such  as  nonsense  words  and  fictitious  content,  in  favor  of  authentic  curricular  materials  from  1993- 
2001.  They  suggest  that  this  trend  is  due  to  the  increased  interest  in  external  validity  and  an  increased  concern  about 
the  applicability  (generalizability)  of  laboratory -controlled  findings  for  real-world  settings. 

An  Examination  of  Internal  and  External  Validity  Considerations 

Ironically,  the  pursuit  of  external  validity  through  non -experimental  methods  inherently  limits  the 
generalizability  of  the  findings.  While  case  studies  and  other  qualitative  methods  are  very  helpful  for  identifying 
potential  pivotal  variables  for  further  study  and  capturing  the  experiential  elements  of  individual  experiences 
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regarding  the  technology  in  question,  they  are  inherently  not  generalizable  beyond  the  time  - and  activity-specific 
setting  in  which  the  data  was  gathered  (Stake,  1998).  Likewise,  the  limitations  of  descriptive  studies  that  utilize 
quantitative  correlational  methods  prevent  researchers  from  validly  concluding  that  use  of  a particular  technology  of 
interest  caused  any  reported  outcome  (Pedhazur  & Schmelkin,  1991). 

Addressing  the  relationship  between  external  validity  and  generalizability,  Banaji  and  Crowder  (1989) 
caution  against  conflating  the  two: 

No  one  would  deny  that,  other  things  being  equal . . .ecologically  valid  methods ...  used  to  achieve 
generalizable  results  is  the  best  situation  in  which  to  find  oneself.  Nor  could  it  possibly  be  denied  that  the 
combination  of  contrived,  artificial  methods  and  conclusions  with  no  external  validity  produces  a sorry 
state....  The  multiplicity  of  uncontrolled  factors  in  naturalistic  contexts  actually  prohibits  generalizability 
to  other  situations  with  different  parameters.  The  implication  that  tests  in  the  real  world  permit  greater 
generalizability  is  false  once  the  immense  variability  from  one  real-world  situation  to  another  is  recognized, 
(pp.  1188-1189). 

In  fact,  they  suggest  that  when  a tradeoff  must  be  made  between  ecological  validity  and  generalizability,  it  is 
generalizability  that  ought  to  win  out,  in  order  to  advance  our  understanding  of  causal  mechanisms  that  can  be 
harnessed  to  develop  technologies.  Ecological  factors,  they  reason,  can  validate  the  generalizability  of  a causal 
mechanism,  but  the  uniquenesses  of  particular  settings  can  obscure  the  accurate  identification  of  causal  principles. 

As  Morton  (1991)  noted  in  his  commentary  on  their  article,  “generalizations  that  do  not  extend  outside  the  restricted 
environment  in  which  they  were  bred  are  not  of  much  use,  irrespective  of  their  beauty”  (p.  33). 

Especially  in  educational  technology  research,  investigators  must  be  cautious  of  sacrificing  internal 
validity,  because  such  a strategy  can  lead  to  confounded  results,  such  as  those  identified  by  Clark  (1983;  1985)  in  his 
discussion  of  media  selection  and  instmctional  strategies.  Clark’s  analysis  of  effect  sizes  in  media  research  revealed 
that  the  most  common  sources  of  invalid  causal  inference  in  technology  studies  were  the  uncontrolled  effects  of  the 
differences  in  the  method  or  content  compared  and  the  novelty  effect  for  the  use  of  a newer  media.  Because  the 
demands  of  authentic  settings  often  require  the  use  of  imperfectly  matched  instructional  materials  (e.g.  lecture  vs. 
interactive  simulation)  or  blatant  introduction  of  new  apparatus,  it  is  often  extraordinarily  difficult  to  design  research 
that  meets  the  demands  of  Banaji  and  Crowder’s  (1991,  p.  78)  “best  possible  situation.” 

Campbell  and  Stanley  (1963)  identified  eight  common  threats  to  internal  validity  that  manifest  as  a design’s 
inability  to  control  the  influence  of  extraneous  variables  (see  Table  1).  These  threats  are  best  controlled  in  the 
laboratory,  where  “true  experiments”  provide  equivalence  of  the  subject  groups  and  tight  controls  over  variables  that 
cannot  be  managed  in  the  field.  However,  to  a great  extent,  carefully  considered  quasi-experimental  designs  can 
also  control  these  potential  confounds.  As  discussed  above,  the  traditional  disadvantage  is  that  the  controlled 
laboratory  environment  often  reduces  the  robustness  of  findings  by  limiting  external  validity.  However,  the  tension 
between  laboratory -based  internal  validity  and  field-based  external  validity  can  be  alleviated  by  creatively 
combining  research  designs  to  measure  the  effects  of  instructional  technology  innovations  (Clark  & Snow,  1975). 

Campbell  & Stanley’s  (1963)  Threats  to  Internal  Validity 

Events,  other  than  the  experimental  treatments,  influence  results. 

During  the  study,  psychological  changes  occur  within  subjects 
Exposure  to  a pretest  or  intervening  assessment  influences  performance  on 
a posttest. 

Testing  instruments  or  conditions  are  inconsistent;  or  pretest  and  posttest 
are  not  equivalent,  creating  an  illusory  change  in  performance. 

Scores  of  subjects  that  are  very  high  or  very  low  tend  to  regress  towards 
the  mean  during  retesting. 

Systematic  differences  exist  in  subjects’  characteristics  between  treatment 
groups . 

Subject  attrition  may  bias  the  results. 

Implementation  of  one  condition  influences  subjects  in  another  condition. 

Past  Attempts  to  Satisfy  Both  Types  of  Validity 

Building  upon  classic  research  methodology,  Clark  and  Snow  (1975)  offered  one  of  the  original  treatises  in 
proposing  alternative  research  designs  specifically  for  instructional  technology.  Common  design  problems  in  the 
studies  they  reviewed  included  reliance  on  pre -experimental  research  designs,  which  lacked  random  assignments  to 
conditions,  control  groups,  and/or  equivalence  among  the  subjects.  Consequently,  it  was  impossible  to  draw  valid 
conclusions  regarding  the  causal  relationships  between  subjects’  characteristics,  technology  design,  and  observed 


Table  1: 

History 

Maturation 

Testing 

Instrumentation 
Statistical  Regression 
Selection 

Experimental  Mortality 
Diffusion  of  Treatments 
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outcomes. 

Unfortunately  these  shortcomings  are  still  in  evidence  today  (Clark  & Estes,  1998,  1999).  Much  of  the 
activity  in  educational  technology,  Clark  agues,  has  been  the  practice  of  craft,  not  technology.  Whereas  technology 
is  the  application  of  scientific  principles  to  solving  real-world  problems  and  the  generalizability  of  solutions,  craft  is 
characterized  more  by  situated  trial  and  error  and  solutions  that  are  indeterminate,  non-transferable,  and 
unconnected  to  a systematic  knowledge  base.  Further,  the  result  of  this  confusion  in  the  practice  of  educational 
technology  is  reflected  in  research  studies  that  report  “no  significant  difference,”  that  fail  to  isolate  the  “active 
ingredient”  for  effect  attribution,  and  that  are  not  generalizable  to  different  contexts. 

Various  alternative  design  solutions  for  balancing  the  conflicting  demands  of  internal  and  external  validity 
have  been  proposed.  Two  designs  that  have  recently  come  to  the  forefront  are  the  randomized  field  experiment  and 
the  design  experiment.  The  randomized  field  experiment  (Ross  & Morrison,  2003)  requires  that  subjects  in  the 
treatment  group  be  randomly  selected  to  eliminate  selection  as  a threat  to  internal  validity.  The  advantage  of  this 
type  of  experiment  is  its  high  external  validity;  however,  internal  validity  with  respect  to  history  and  the  overall 
complexity  of  the  experiment  allows  for  confounding  variables. 

Design  experiments  (Winn,  2003)  are  also  conducted  mostly  in  field  environments.  In  this  design,  the 
treatment,  such  as  an  instructional  tool  or  strategy,  is  applied  in  an  educational  setting,  and  data  is  gathered.  Based 
on  an  analysis  of  the  data,  the  treatment  is  revised  and  applied  with  additional  data  collection.  This  iterative 
intervention  continues  over  time  until  the  treatment  is  proved  consistently  effective.  It  is  the  iterative  nature  of 
design  experiments  that  controls  for  spurious  variables,  in  that  the  intervention  can  be  adapted  to  correct  a problem 
that  the  analysis  of  the  data  reveals.  Design  experiments  should  be  replicated  to  establish  validity;  however,  “the 
techniques  for  gathering  and  analyzing  data  in  design  experiments  are  typically  less  prescribed  and  less  procedural 
than  those  used  in  experimental  studies”  (p.  370). 


A New  Design 

Although  conducted  in  real  world  settings,  randomized  field  experiments  and  design  experiments  radically 
tip  the  balance  of  the  evidence  collected  in  favor  of  external  validity.  Due  to  the  threats  to  internal  validity  inherent 
in  both  designs  through  history,  maturation,  and  diffusion  of  treatment,  the  results  of  studies  using  these  and  similar 
designs  may  only  be  useful  to  generally  interpret  learning  outcomes,  rather  than  assist  researchers  to  understand  the 
specific  mechanisms  underlying  and  supporting  student  learning.  It  is  critical,  therefore,  to  continue  to  refine 
experimental  designs  in  educational  technology  research  to  improve  the  yield  of  experiments  in  complex  field 
settings,  the  causal  inferences  they  provide,  and  the  generalizability  of  these  inferences  to  constructs  over  a variety 
of  populations,  settings,  treatments,  and  outcomes  (Shadish,  Cook,  & Campbell,  2002). 

In  the  case  of  distance  learning,  the  balance  between  internal  and  external  validity  must  be  informed  by  the 
realities  of  its  implementation.  In  addition  to  taking  place  over  a geographically  distributed  area  at  varying  times, 
course  designs  are  often  modular  in  nature.  That  is,  instructional  content  is  divided  into  sequential  modules  that 
deliver  the  curricular  content  in  a logical  order  (Khan,  2001).  Because  some  content  is  often  dependent  on  the 
mastery  of  material  presented  in  a previous  module,  consideration  of  such  features  is  vital  during  the  experimental 
design  process  to  preserve  both  internal  validity  (i.e.  eliminate  history  and  maturation  threats)  and  external  validity 
(i.e.  maintain  authentic  delivery). 

Built  specifically  with  sequential  module -based  courses  in  mind,  the  Strand  of  Pearls  (SOP;  see  Figure  1) 
design  consists  of  four  conditions  into  which  distance  education  subjects  are  randomly  placed.  Each  condition 
presents  the  course  material  in  the  same  sequence  and  with  the  same  timing.  What  differs  among  the  conditions  for 
each  module  is  solely  the  inclusion  or  exclusion  of  the  targeted  feature  to  be  tested.  The  first  condition  entails  a 
standard  sequence  of  instructional  modules  that  do  not  include  the  feature  to  be  tested  as  an  equivalent  control  group 
against  which  treatment  modules  are  compared  (Group  A in  Figure  1).  As  with  a conventional  untreated  control 
group  design,  the  second  condition  into  which  subjects  may  be  assigned  delivers  each  module  in  the  same  sequence 
as  the  first  group,  though  every  module  includes  the  experimental  feature  to  be  evaluated  (Group  D in  Figure  1). 

The  third  and  fourth  conditions  represent  systematic  alternating  sequences  of  the  experimental  and  the  control 
versions  of  the  modules  (Groups  B and  C in  Figure  1).  The  difference  between  the  third  and  fourth  conditions  lies 
in  the  sequencing  of  the  experimental  and  control  modules.  In  Group  B,  the  first  quarter  of  the  modules  in  the 
course  are  control  versions,  the  second  quarter  are  experimental  versions,  and  so  on.  The  order  for  Group  C is  a 
reverse  of  the  B,  so  that  the  first  quarter  of  the  modules  use  the  experimental  version,  the  second  quarter  use  the 
control  modules,  etc.  In  this  way,  half  of  the  subjects  across  all  conditions  are  using  the  experimental  modules  at 
any  point  in  the  course. 
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In  each  module  where  it  is  present,  the  feature  is  implemented  in  precisely  the  same  way  in  order  to 
maintain  the  comparability  of  effects  both  longitudinally  within  groups  and  cross  -sectionally  across  modules.  By 
alternating  the  availability  of  the  feature  across  two  of  the  four  groups,  it  becomes  possible  to  detect  any  confounds 
or  threats  to  internal  validity  due  to  time  enrolled  in  the  course  (history,  maturation),  widespread  environmental 
influences  (history),  and  aberrant  results  due  to  an  interaction  between  the  feature  tested  and  module -specific 
content. 

In  order  to  achieve  external  validity,  the  features  of  instructional  delivery  must  be  consistent  between  actual 
and  experimental  use.  Such  elements  include  the  settings  in  which  subjects  participate  in  the  course,  the 
technological  impediments  that  might  impact  the  use  of  technological  features,  the  presentation  sequence  and 
content  of  the  curriculum,  and  the  assessment  mechanisms.  Additionally,  subjects’  attention  and  actions  must  not  be 
unduly  influenced  by  awareness  of  the  fact  that  they  are  participating  in  an  experiment  (i.e.  the  Hawthorne  effect). 
Each  of  these  elements  is  satisfied,  because  the  study  occurs  through  the  vehicle  of  the  course  delivery  itself.  As 
such,  all  of  the  user  experiences  are  authentic  and  limit  the  visibility  of  the  research  being  conducted. 

Further,  generalizability  is  also  preserved,  because  the  sole  consistent  difference  between  a particular 
module  used  by  the  experimental  groups  and  control  groups  is  the  availability  of  the  instructional  tool  being  tested. 
Thus,  the  findings  will  predominantly  reflect  only  that  manipulation.  Typically,  one  of  the  major  confounds  in  field 
studies  is  the  environment  in  which  each  subject  is  participating.  In  classrooms  during  live  instruction  or  in  other 
scenarios  in  which  subjects  are  in  a particular  physical  setting,  there  are  many  uncontrolled  commonalit  ies  (e.g. 
culture,  socio-economic  status,  teacher  bias,  etc.).  However,  in  distance  learning  scenarios,  such  environmental 
influences  are  limited,  because  the  variance  introduced  by  environmental  effects  on  a single  subject  will  not  have  a 
statistically  significant  impact  on  the  overall  results,  if  sample  sizes  are  sufficient  to  provide  appropriate  statistical 
power.  Further,  it  can  be  argued  that  those  environmental  variables  consistent  enough  across  users  to  generate  an 
overall  effect  are  common  to  the  larger  population  of  distance  learners. 

An  additional  evaluative  approach  to  enhancing  generalizability  leverages  the  sequential  and  modular 
design  of  most  distance  learning  courses.  By  transparently  evaluating  subjects’  knowledge  levels  at  frequent 
intervals,  using  pre-  and  post-test  assessments  between  each  module,  longitudinal  data  is  acquired  for  each  subject 
that  can  be  used  to  detect  statistical  outliers  within  individuals  over  time,  in  addition  to  particular  subjects  relative  to 
the  sample  population.  Thus,  if  a subject  generates  aberrant  data  for  a particular  module  (possibly  for  reasons  of 
environmental  interference  that  would  not  be  representative  of  the  general  population),  that  data  can  be  withheld 
from  overall  analyses  to  increase  the  generalizability  of  the  findings. 
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The  advantage  of  this  design  over  other  research  paradigms  is  that  it  combines  the  external  validity  of  a 
staged  innovation  design  (Clark  & Snow,  1975)  with  the  internal  validity  of  an  equivalent  time  samples  design  (Best 
& Kahn,  2003).  The  strength  of  the  staged  innovation  design  is  that  the  “control’'  and  “experimental’'  groups  are 
created  by  time  -shifting  the  treatment;  however,  this  is  also  its  weakness  for  internal  validity,  especially  with  respect 
to  history.  The  traditional  equivalent  time  samples  designs,  on  the  other  hand,  minimize  the  effect  of  history; 
however,  they  increase  the  strength  of  potential  confounds,  such  as  maturation,  unstable  instrumentation,  testing, 
and  experimental  mortality.  In  the  proposed  design,  the  four  conditions,  considered  collectively,  serve  as  counter- 
balanced controls  to  evaluate  not  only  the  net  effect  of  the  experimental  feature,  but  also  the  intraindividual  effects 
on  the  rate  and  outcome  of  skill  development  related  to  the  module  design  being  tested. 

An  Example  of  Application 

To  illustrate  the  advantages  of  the  Strand  of  Pearls  design,  we  will  consider  the  case  of  a distance  learning 
course  in  U.S.  law  for  foreign  students  taken  in  their  home  country.  Historically,  such  students  have  either  received 
conversational  English  language  instruction  prior  to  the  course  or  enroll  in  a conversational  English  language  course 
concurrent  with  the  law  course.  However,  recent  studies  have  indicated  that  these  approaches  do  not  adequately 
prepare  these  students  to  succeed  in  advanced  courses  (Brostoff,  Sinsheimer,  & Ford,  2001;  Feak  & Reinhart,  2002; 
Hanigsberg,  1994).  As  the  language  of  law  and  legal  studies  is  complex  and  difficult,  these  students  might  benefit 
from  language  instruction  embedded  in  their  law  courses  to  master  the  content  objectives  and  language  learning 
goals  simultaneously. 

In  such  courses,  students  must  be  able  to  read,  summarize,  and  analyze  previously  adjudicated  cases  from 
the  U.S.  Supreme  Court  and  various  Courts  of  Appeal  to  learn  how  to  identify  the  legal  principle  at  issue,  the  facts 
of  the  case  under  consideration,  and  the  reasoning  the  court  used  to  reach  its  finding.  Both  the  density  of  the 
language,  often  comprised  of  long  sentences  containing  multiple  conditional  clauses  and  the  organizational  intricacy 
of  the  case  often  defeat  students’  attempts  to  extract  the  necessary  information. 

One  method  to  enhance  the  readability,  comprehension,  and  retention  of  complex  text  is  the  use  of  textual 
cues  (Kerper,  2001).  Textual  cues  are  added  to  the  reading  text  by  reformatting  the  page  to  create  additional  white 
space,  allowing  the  placement  of  “textual  cues”  in  the  form  of  headings  or  questions.  These  cues  direct  the  reader’s 
attention  to  a particularly  important  fact  or  issue  that  is  essential  for  understanding  the  reading  assignment. 

In  our  example,  the  purpose  of  our  study  is  to  determine  the  effect  of  the  treatment,  in  this  case,  the  use  of  textual 
cues  in  reading  assignments,  on  students’  achievement  of  the  course  objectives,  with  the  corresponding  null 
hypothesis  that  the  treatment  has  no  effect  on  student  achievement.  As  each  course  module  contains  the  text  of  one 
complete  case  taken  from  the  body  of  U.S.  case  law,  the  achievement  of  the  individual  module’s  learning  objectives, 
and  thus  the  overall  course  goals,  is  dependent  on  the  understanding  of  each  module’s  case.  As  such,  the  course  is 
an  ideal  mechanism  for  measuring  the  effectiveness  of  this  proposal  and  for  testing  our  hypothesis  by  helping  to 
isolate  the  respective  influences  of  language  and  skill  proficiency  on  outcome  assessments. 

The  course  consists  of  8 ordered  modules  of  computer-based  instruction.  Each  module  incorporates 
multimedia,  the  court  case,  exercises,  and  self-assessments.  Two  versions  of  each  court  case  are  prepared,  one 
version  that  incorporates  textual  cues,  and  another  version  in  which  the  text  remains  in  its  original  form.  The 
study’s  control  and  experimental  conditions  are  solely  determined  by  which  version  of  the  case  text  is  present  in  the 
module.  Except  for  this  intervention,  all  other  module  components  are  identical  in  content,  method,  and 
instructional  sequence  in  the  control  and  experimental  conditions. 

The  course  is  delivered  to  students  via  an  Internet  Web  site.  Prior  to  commencing  the  course,  participants 
complete  a questionnaire  to  collect  background  data  with  respect  to  age,  level  of  education,  experience,  and  English 
language  fluency.  In  addition,  pretests  to  measure  knowledge  of  U.S.  law  and  to  assess  English  language  skills  are 
administered  to  the  participants. 

Following  the  SOP  design,  participants  in  the  study  are  randomly  assigned  to  one  of  the  four  conditions. 

The  control  condition  (Group  A in  Figure  1)  provides  instruction  in  law  only  and  does  not  include  the  textual 
treatment,  and  the  experimental  condition  (Group  D in  Figure  1)  includes  the  textual  treatment.  As  described  above, 
the  third  and  fourth  conditions  (Groups  B and  C in  Figure  1)  alternate  between  the  language  treatment-included  (TI) 
and  treatment-excluded  (TX)  modules  in  two -module  segments  (i.e.,  TI-TX-TI-TX  and  TX-TI-TX-TI  sequences). 
After  each  module,  a combined  posttest  of  the  previous  module  and  pre-test  of  the  subsequent  model  is  completed 
by  each  participant.  With  the  use  of  module -specific  and  overall  pre-  and  posttests,  a rich  data  pool  can  be  analyzed 
with  great  confidence  in  both  internal  and  external  validity. 

Longitudinal  Analyses  Across  Modules.  For  instance,  it  is  conceivable  that  a direct  comparison  of  the 
achievement  data  for  Group  A and  Group  D following  completion  of  all  8 modules  might  reveal  no  significant 
difference  in  the  effect  of  the  experimental  treatment,  initially  indicating  a failure  to  reject  the  null  hypothesis. 
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However,  a longitudinal  analysis  of  the  posttest  data  for  each  module  might  demonstrate  a significant  effect  early  in 
the  course  sequence  that  diminished  as  the  course  progressed.  As  initial  successes  could  be  expected  to  beneficially 
impact  academic  motivation  and  early  course  attrition,  such  data  could  be  used  tojustify  the  use  of  the  treatment 
only  in  initial  modules — a finding  that  would  have  been  overlooked  in  a conventional  summative  comparison. 

An  inter- module  analysis  of  the  data  for  Groups  B and  C is  also  of  interest.  Following  completion  of 
Modules  1 and  2,  Groups  B and  C might  demonstrate  significant  effect  differences.  If  the  differences  in  these  same 
groups  following  completion  of  Modules  3 and  4 were  similar  but  reversed , and  this  trend  continued  in  an 
alternating  dyad  sequence,  the  strength  of  the  hypothesis  in  favor  of  a positive  treatment  effect  would  be  increased. 

Another  inter- module  analysis  might  examine  the  data  within  any  of  the  four  conditions  following  the 
completion  of  each  module.  This  analysis  would  focus  on  the  trends  established  by  the  data.  For  example,  the  slope 
of  a graph  representing  post-module  achievement  scores  for  Group  A might  be  compared  with  the  slope  for  Group  B 
to  examine  the  achievement  rate  over  time  and  any  possible  cumulative  effects  of  the  treatment,  whether  positive  or 
negative. 

Cross-Sectional  Analyses  Within  Modules.  Controlling  for  variables  resulting  from  the  background  data, 
the  SOP  design  enables  cross-sectional  analyses  within  modules  to  detect  indications  of  threats  to  internal  validity 
and  anomalous  results.  Modular  anomalies  can  be  isolated  and  examined  further  to  determine  possible  causes.  For 
example,  if  the  treatment  were  to  be  effective  only  with  certain  modules  and  not  others,  additional  qualitative 
analyses  could  be  performed  to  develop  possible  explanations,  such  as  unforeseen  interactions  between  the  content 
specific  to  those  modules  and  the  treatment  (e.g.  assessment  for  a particular  module  emphasizes  non-linguistic 
information,  thereby  reducing  the  effectiveness  of  the  language-oriented  treatment).  Further,  other  threats  to  internal 
validity,  such  as  maturation  or  interference  from  history,  might  be  identified. 

Because  the  course  serves  both  instructional  and  experimental  purposes,  external  validity  is  maximized,  in 
that  the  content,  modular  sequence,  technology,  and  assessments  are  consistent  in  both  actual  and  experimental  use. 
Additionally,  participants  take  part  in  the  course  on  an  individual  basis  at  a distance  via  the  Internet  and  therefore, 
have  no  contact  with  one  another,  which  protects  against  the  diffusion  of  treatment  threat.  Further,  subjects  need  not 
be  aware  of,  or  influenced  by,  the  experimental  nature  of  the  course. 

Conclusion 

Distance  learning  courses  provide  a unique  opportunity  for  research,  where  the  internal  validity  of  the 
laboratory  and  the  external  validity  and  generalizability  of  the  field  can  be  called  upon  to  support  the  conclusions  of 
instructional  treatment  studies.  The  inherently  useful  state  of  modularity  common  to  these  courses  provides  a 
practical  tool  for  isolating  treatments  under  randomized  and  controlled  conditions.  In  particular,  the  Strand  of  Pearls 
design  facilitates  a variety  of  cross-sectional  and  longitudinal  analyses  that  aid  researchers  in  establishing  robust 
convergent  evidence  for  the  effectiveness  of  particular  treatments.  Through  such  efforts,  it  is  our  hope  that  the 
“active  ingredients”  of  effective  learning  can  be  identified  and  leveraged  to  improve  both  the  scientific 
understanding  of  human  learning  and  the  quality  of  instruction  available  to  learners  across  the  globe. 
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Novice  Learners  in  Cyberspace 
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Since  the  Internet  was  introduced  in  the  field  of  Educational  Technology,  it  has  predominantly  been  seen  as 
an  instructional  tool.  A number  of  educational  technologists  point  out  that  the  Internet  can  be  a powerful 
instructional  tool  that  encourages  learners  to  be  involved  in  problem-solving  by  using  a great  deal  of  information 
and  to  enlarge  and  deepen  their  knowledge  by  sharing  their  opinions  through  a variety  of  communication  tools  such 
as  emails,  thrilled  discussios  and  chat  rooms  (Harasim,  Calvert,  & Groeneboer,  1997;  Herrington  & Oliver, 
2000;Jonassen  & Hernandez-Serrano,  2002;  Khan,  1997;  McCormack  & Jones,  1998;  Ritchie  & Hoffman,  1997; 
Romiszowski,  1997).  These  researchers  have  studied  how  teachers  can  use  the  Internet  to  help  learners  construct 
meaningful  learning  for  problem-solving  in  real  life  context  and  to  dlow  them  to  be  engaged  in  learning.  In 
addition,  they  have  done  research  on  how  to  create  useful  and  effective  learning  environments  including  design  of 
specific  contents,  development  of  instructional  strategy,  and  educational  evaluation  using  the  Internet.  The 
popularization  of  the  term  web-based  instruction  shows  how  many  researchers  and  educators  think  of  the  Internet  by 
as  tool  for  teaching  and  learning. 

This  paper,  however,  raises  another  perspective  on  the  Internet  in  the  field  of  Educational  Technology, 
considering  it  as  a cultural  space.  Cyberspace,  first  discussed  in  a science  fiction  novel  by  Gibson  (1984),  is  an 
appropriate  term  to  describe  the  space  of  this  perspective.  In  general,  cyberspace  refers  to  the  interactive  digital 
space  created  by  computer  networks,  in  particular  the  Internet  (Mann  & Stewart,  2000).  Many  researchers  and 
scholars  argue  that  beyond  an  instructional  tool,  the  Internet  provides  a cultural  space  where  people  have 
interactions,  communicate  with  each  other,  and  build  online  communities  (Bell,  2001;  Bell  & Kennedy,  2001; 
Correll,  1995;  Hine,  2000;  Markham,  1998;  Turkle,  1995).  These  researchers  assert  that  cyberspace  has  an 
influence  on  the  way  people  think,  behave,  feel,  and  communicate  with  others  in  terms  of  race,  ^nder,  class, 
religion,  etc.  In  addition,  they  point  out  that  people  in  cyberspace  represent  themselves  in  certain  ways  that  might 
not  resemble  how  they  represent  themselves  in  physical  spaces2.  In  this  light,  Turkle  (1995)  states  that  cyberspace 
is  part  of  people’s  everyday  life.  Thus,  in  this  perspective  on  the  Internet,  cyberspace  as  a cultural  space,  human 
interaction,  recognition  of  selves  and  others,  and  people’s  experiences  can  be  centered  in  cyberspace 

Given  that  cyberspace  is  a cultural  space,  it  is  obvious  that  learning  occurs  in  cyberspace  and  that  it  is 
embedded  in  social  and  cultural  contexts  in  which  learning  occurs.  While  there  are  many  studies  on  cyberspace  as  a 
cultural  space,  a majority  of  these  studies  have  concerned  online  identity  or  online  community,  not  educational 
environments.  Little  research  on  learning  in  cyberspace  has  been  conducted.  As  large  numbers  of  students  come  to 
be  involved  in  learning  in  cyberspace  at  university  and  K-12  levels,  it  is  necessary  to  understand  how  students  learn 
in  cyberspace  when  they  cannot  see  their  teachers  and  other  students  face-to-face. 

As  many  social  theorists  argue  for  physical  space,  we  need  to  take  a social  theory  of  learning  for  examining 
learning  in  cyberspace.  Situated  learning,  a well-known  social  theory  of  learning  by  Lave  and  Wenger  (1991),  can 
be  an  appropriate  theoretical  lens  for  this  examination.  According  to  Lave  and  Wenger,  social,  cultural,  and 
historical  interactions  always  bring  about  situated  activities  or  situated  learning.  Given  that  cyberspace  is  a cultural 
space  based  on  communities  and  cultures,  I argue  that  learning  in  cyberspace  is  situated  learning  that  leads  novice 
learners  to  participate  fully  in  the  community  of  practice  through  the  legitimate  peripheral  participation.  In  this 
light,  it  is  of  importance  to  understand  how  novice  learners  become  part  of  the  community  in  cyberspace. 

Since  situated  learning  in  cyberspace  occurs  in  social  and  cultural  space,  a cultural  studies  approach 
focusing  on  discourse,  subjectivity,  and  agency  is  useful  for  understanding  learning,  knowledge,  and  power  in 
cyberspace.  As  Lave  and  Wegner  (1991)  argue,  learning  and  knowledge,  whether  in  physical  spaces  or  in 
cyberspace,  is  always  related  with  power  relationship  between  novice  learners  and  experts  (eg.  teachers).  In  other 
words,  discourses  of  both  novice  learners  and  experts,  novice  learners’  multiple  and  fluid  subjectivities,  and  the  way 
novice  learners  excise  their  agency  in  a certain  extent  and  condition  can  influence  the  negotiations  and  constructions 
of  meaning  in  cyberspace  learning  contexts. 

Thus,  the  purpose  of  this  qualitative  study  is  to  investigate  how  novice  learners  become  part  of  a cyber 
community  through  the  legitimate  peripheral  participation  in  cyberspace  in  terms  cf  discourse,  subjectivity,  and 
agency.  This  paper  addresses  an  issue  of  situated  learning  in  cyberspace  using  a cultural  studies  approach  that  will 
include  the  data  collection  process  and  a preliminary  analysis  of  data. 
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Conceptualization  of  Cyberspace  as  a Cultural  Space 

The  term  cyberspace  is  not  an  easy  word  to  define  in  that  it  describes  a virtual  world  that  is  mediated  by  a 
computer  network.  Due  to  this  uniqueness,  cyberspace  is  conceptualized  in  a variety  of  ways.  Beneditk  (2000) 
defines  cyberspace  as  “a  world  in  which  the  global  traffic  of  knowledge,  secrets,  measurements,  indicators, 
entertainments,  and  alter-human  agency  takes  on  form:  sights,  sounds,  presences  never  seen  on  the  surface  of  the 
earth  blossoming  in  a vast  electronic  light”  (p.29).  On  the  other  hand,  he  also  describes  cyberspace  as  “a  common 
mental  geography,  built,  in  turn,  by  consensus  and  revolution,  canon,  experiment;  a territory  swarming  with  data  and 
lies,  with  mind  stuff  and  memories  of  nature,  with  a million  voices  and  two  million  eyes  in  a silent,  invisible  concert 
to  enquiry,  deal-making,  dream  sharing,  and  simple  beholding”  (p.29).  His  conceptualization  of  cyberspace  implies 
that  cyberspace  is  not  only  a simple  space  created  by  machine,  computer  network,  but  also  a concrete  place  where 
millions  of  people  exist,  stay  for  a while,  and  live  for  a certain  time  of  their  lives  with  active  human  interactions. 

Bell  (2001)  makes  this  broad  and  complicated  conceptualization  of  cyberspace  much  clearer.  He 
conceptualizes  cyberspace  as  a cultural  space  that  is  lived  and  made  from  people,  machine,  and  stories  in  everyday 
life.  He  defines  cyberspace  by  using  three  modes  of  story-telling.  First,  he  defines  cyberspace  in  terms  of  hardware 
that  facilitates  a form  of  interaction  between  remote  actors,  which  calls  material  stories.  As  an  alternative  definition, 
he  defines  cyberspace  as  an  imagined  space  between  computers  in  which  people  might  build  new  selves  and  new 
worlds,  which  he  calls  symbolic  stories.  According  to  Bell,  cyberspace  k all  this  and  more.  In  order  words, 
cyberspace  is  hardware  and  software  and  “image  and  ideas”  (p.7).  In  this  light,  these  two  stories  are  inseparable. 
Moreover,  Bell  argues  that  the  ways  we  experience  cyberspace  represent  a negotiation  of  material  and  symbolic 
elements,  which  he  calls  experiential  stories.  In  Bell’s  definition,  human  interaction,  recognition  of  selves  and 
others,  and  people’s  experiences  are  centered  in  cyberspace.  In  this  light,  he  emphasizes  that  cyberspace  is  always 
cyberculture  in  that  any  and  every  thing  around  us  is  the  product  of  culture. 

Turkle  (1995)  has  a similar  point  of  view  to  Bell.  She  sees  cyberspace  as  a cultural  space  of  simulation  in 
that  even  though  they  might  not  see  others,  people  have  the  opportunity  to  build  new  kinds  of  communities  in  which 
they  participate  with  others  from  all  over  the  world,  others  with  whom  they  have  conversations  everyday,  or  others 
with  whom  they  may  have  deeply  intimate  relationships.  In  this  regard,  like  Bell,  she  focuses  on  people’s 
interaction  and  experiences  in  cyberspace  and  believes  that  cyberspace  provides  people  with  a new  environment  for 
social  and  cultural  interaction.  Differing  from  Bell,  however,  she  emphasizes  specific  local  contexts  in  cyberspace. 
In  the  explanation  of  constructing  identity  in  cyberspace,  she  argues  that  experiences  in  cyberspace  can  only  be 
understood  in  the  larger  cultural  context. 

In  sum,  cyberspace  is  not  only  a virtual  space  mediated  by  computer  networks  but  also  is  conceptualized  as 
a cultural  space  in  which  human  interaction  takes  place,  social  and  cultural  communities  are  built,  and  human 
experiences  are  situated  in  the  social  and  cultural  context  in  which  they  work. 

Situated  Learning  in  Cyberspace 

Situated  learning  is  based  on  an  assumption  that  learning  is  constructed  by  the  individual’s  negotiation  of 
meaning  in  a specific  social  and  cultural  context  (Lave  & Wegner,  1991:  Wenger,  1998).  Since  cyberspace  is  a 
cultural  space,  I argue  that  learning  in  cyberspace  is  situated  learning  and  conceptualize  cyberspace  as  the  site  of 
situatedness.  In  other  worlds,  borrowing  from  Lave  and  Wenger’s  (1991)  argument  on  situated  learning,  I argue  that 
there  is  no  activity  in  cyberspace  that  is  not  situated.  Through  negotiation  of  meaning,  learning  in  cyberspace  is  an 
integral  part  of  generative  social  practice  in  the  lived-in  world.  Consequently,  novice  learners  in  cyberspace  will 
become  experts  through  legitimate  peripheral  participation  through  the  community  of  practice  in  cyberspace.  Thus, 
the  assumption  of  situated  learning  fits  into  learning  in  cyberspace  in  that  any  novice  learners’  activity  in  cyberspace 
is  situated  in  a given  context  and  thus,  needs  to  negotiate  meaning  with  their  experts  through  written  text. 

As  the  findings  of  many  studies  show  (Hine,  2000;  Markham,  1998;  Turkle,  1995),  cyberspace  provides 
people  with  online  communities  in  which  they  spend  a great  deal  of  time,  they  see  what  is  happening,  they  exchange 
their  ideas,  they  learn,  and  they  practice  what  they  learn.  In  this  light,  it  can  be  said  that  online  communities  are 
communities  of  practice  for  situated  learning  (Hung  & Chen,  2002).  While  rainy  studies  on  communities  of 
practice  in  cyberspace  have  begun,  almost  all  communities  of  practice  in  the  studies  are  non -educational 
communities. 

Although  Lave  and  Wenger  (1991)  contend  that  schooling  cannot  be  exactly  situated  learning  because  its 
characteristic  is  institutional,  many  scholars  seem  to  consider  that  schooling  is  situated  learning  in  that  every  activity 
is  situated  (Brown,  Collins,  & Duguid,  1989;  Hung,  2001).  They  see  classroom  environments  as  communities  of 
practice  that  allow  learners  to  become  part  of  communities  in  terms  of  learning  topics  or  issues.  Consistent  with  this 
view,  it  is  possible  to  think  of  teaching  and  learning  processes  in  cyberspace  as  constructing  communities  of 
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practice. 

Online  middle  and  high  schools  can  be  a good  example  of  an  educational  community  in  cyberspace.  In 
general,  online  schools  provide  students  with  qualified  teachers  and  state-wide  approved  learning  content.  Teachers 
and  students  at  online  schools  have  a continuous  bond  and  interactions  for  an  academic  year  in  an  online  community 
even  though  they  do  not  need  to  meet  each  other  face-to-face.  Like  novices  in  physical  places,  novice  students  who 
start  going  to  online  schools  for  the  first  time  need  to  negotiate  both  the  meaning  of  learning  and  the  online 
community  culture  where  they  belong  to  in  a situated  context.  When  they  negotiate  these  with  their  teachers  as 
experts  in  cyberspace,  as  Lave  and  Wenger  (1991)  argue,  there  are  always  power  relationships  between  novice 
learners  and  teachers  that  affects  learners’  meaningful  learning.  These  power  relationships  should  be  understood 
and  analyzed  in  a specific  local  and  cultural  context. 

A Cultural  Studies  Approach  to  Situated  Learning  in  Cyberspace 

Cultural  studies  differs  from  traditional  academic  disciplines  having  definite  areas  for  investigation  and  its 
own  particular  methodology  in  that  a variety  of  disciplines  such  as  philosophy,  psychoanalysis,  sociology, 
anthropology,  history,  language,  semiotics,  poststructuralism,  postmodernism,  and  feminism  have  had  a great  deal  of 
influence  on  cultural  studies  (During,  1993;  Grossberg,  Nelson,  & Treichler,  1992).  While  some  researchers  call 
cultural  studies  interdisciplinary,  other  theorists  consider  cultural  studies  anti-disciplinary  in  that  it  is  hardly  easy  to 
define  it  in  terms  of  academic  disciplines.  Rather  than  finding  its  simple  origin,  cultural  studies  tends  to  be 
understood  as  “a  discursive  formation”  (Hall,  1993,  p.98).  Cultural  studies  has  focused  on  a political  perspective  in 
order  to  articulate  how  dominant  groups  manipulate  subordinate  ones  within  culture.  Most  cultural  studies  theorists 
believe  that  every  social  meaning  and  knowledge  production  is  based  on  power  relationships. 

In  the  field  of  cultural  studies,  discourse,  subjectivity,  and  agency  can  be  core  concepts  for  understanding 
the  connection  between  social  relations  and  meanings.  Discourse  is  defined  as  not  only  language-in-use  in  everyday 
life  but  also  a system  that  has  its  own  rules  and  constraints,  produces  meaning  through  practice  in  social,  cultural, 
and  historical  contexts  (Foucault,  1972,  1978).  Language  creates  a perspective  by  which  human  subjects  make 
sense  of  the  world.  Each  perspective  imposed  by  language  requires  human  subjects  to  negotiate  meanings  in  social 
and  cultural  contexts,  which  is  always  political  (Gee,  1999).  This  political  characteristic  of  language  leads  cultural 
studies  theorists  to  pay  attention  to  discourse  in  terms  of  learning,  knowledge,  and  power.  Subjectivity  refers  to  the 
positions  of  the  individuals  (subjects)  within  a particular  discourse.  In  other  words,  subjectivity  can  be  defined  as 
locating  one  in  a particular  social  position.  Subjectivity  is  crucial  in  cultural  studies  in  that  it  connects  identity 
existing  in  individual’s  mind  with  social  and  cultural  contexts.  Agency  means  capability  of  people  to  think, 
determine,  and  act  autonomously.  Some  forms  of  agency  can  be  the  finding  of  new  directions  while  others  can  be 
resistance  and  struggle.  Agency  can  be  considered  as  the  most  important  issue  in  cultural  studies  because  the  most 
critical  concerns  of  cultural  studies  is  the  ways  in  which  culture  shapes  human  action  (Smith,  2001).  The  three 
concepts  of  discourse,  subjectivity,  and  agency  are  interrelated  with  each  other  rather  than  discrete. 

Taking  the  heavy  reliance  on  written  texts  in  cyberspace  into  account,  it  can  be  said  that  discourse  has  more 
effect  on  situated  learning  in  cyberspace  than  that  in  physical  spaces  in  that  novice  learners  and  experts  must 
represent  their  every  activity  as  written  texts.  This  unique  type  of  discourse  in  cyberspace  can  have  different 
influences  on  people  and  their  situated  learning.  Some  people  may  feel  comfortable  in  using  written  texts  for  their 
learning  and  some  other  people  may  not.  In  addition,  the  written  discourse  in  cyberspace  allows  people  to  observe 
and  learn  how  to  write,  how  to  behave,  how  to  respond,  etc.  In  this  light,  it  can  be  said  that  discourse  in  cyberspace 
has  similar  characteristics  to  that  in  physical  spaces.  Thus,  it  is  necessary  to  investigate  how  novice  learners  and 
experts  have  an  influence  on  the  learners’  negotiation  for  meaningfulness  in  cyberspace  learning  contexts. 

Like  that  in  physical  spaces,  discourse  in  cyberspace  has  an  influence  on  subjectivity.  In  addition  to  this 
issue,  cyberspace  can  allow  people  to  posit  their  subject  position  falsely  on  purpose  because  of  invisibility.  This 
situation  can  frequently  happen  in  online  communities.  However,  in  a school-based  online  community,  it  is 
expected  that  students  and  teachers  tend  to  expose  their  subjectivity  without  deception.  Since  there  are  few  studies 
on  subjectivity  in  situated  learning  in  cyberspace,  it  is  necessary  to  examine  how  novice  learners’  multiple  and  fluid 
subjectivities  influence  negotiation  for  meaningfulness  in  cyberspace  learning  contexts. 

Regarding  agency,  due  to  the  characteristics  of  invisibility  and  heavy  reliance  on  written  texts,  cyberspace 
can  allow  people  to  act,  determine,  and  represent  in  different  ways  from  physical  spaces.  Novice  learners’  agency  in 
negotiating  meaning  in  cyberspace  can  differ  from  that  in  physical  spaces.  In  this  light,  it  can  be  important  to  do  an 
inquiry  on  how  novice  learners  become  agents  overcoming  experts’  dominant  power  through  legitimate  peripheral 
participation  in  cyberspace  learning  contexts. 
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Method 


The  Interpretive  Paradigm 

This  study  is  based  on  the  interpretive  paradigm  because  I focus  on  the  understanding  how  novice  learners 
negotiate  meaning  in  terms  of  their  discourse,  subjectivity,  and  agency.  Unlike  positivism,  the  interpretive  paradigm 
assumes  that  there  are  multiple  realities  that  are  socially  and  experientially  constructed  in  specific  and  local  contexts 
(Guba  & Lincoln,  1994).  These  realities  come  from  human  interactions  for  meaning  making.  Consistent  with  this, 
the  interpretive  paradigm  assumes  that  knowledge  consists  of  “the  reconstruction  of  intersubjective  meanings” 
(Greene,  1990,  p.235).  Knowledge  of  a topic  or  an  issue  is  not  unique  but  plural  because  multiple  reconstructions  of 
meaning  are  possible.  In  addition,  knowledge  is  value-added  rather  than  neutral  since  a viewpoint  on  knowledge 
can  be  different  based  on  race,  gender,  class,  ethnicity,  sexuality,  or  religion  as  well  as  individual’s  belief  or  value. 
Thus,  the  findings  of  investigations  are  created  (Lincoln  & Guba,  2000)  rather  than  discovered.  In  consequence,  the 
interpretive  paradigm  purses  the  interpretive  understanding  of  the  meanings  people  construct  in  a local  and  specific 
context  and  how  these  meanings  interrelated  with  each  other  for  making  them  integrated. 

The  ontology  and  epistemology  of  the  interpretive  paradigm  lead  interpretive  researchers  to  pay  attention  to 
a cultural  space  embedded  in  power  relations  such  as  cyberspace  as  well  as  physical  space  because  they  believe  that 
“social  reality  is  defined  ty  the  nature  and  distribution  of  domination,  power,  and  influence”  (Mezirow,  1996, 
p.  161).  With  attention  to  culture,  the  interpretive  paradigm  tends  to  focus  on  discourse  that  has  an  effect  on  people’s 
subjectivity  and  agency  in  everyday  life  in  terms  of  social,  cultural,  and  historical  aspects.  In  this  light,  it  argues  that 
one  must  gain  an  access  to  culture  and  participate  in  it  for  an  in-depth  understanding  of  nature  of  reality  and 
knowledge.  This  argument  can  provide  an  ethnographic  qualitative  approach  in  terms  of  methodology  of  the 
interpretive  paradigm. 

An  Ethnographic  Qualitative  Research 

My  study  on  situated  learning  in  cyberspace  is  based  on  culture.  Even  though  it  does  not  allow  novice 
learners  and  experts  to  see  each  other  face-to-face,  in  general,  cyberspace  provides  them  with  social  and  cultural 
interactions,  allows  them  to  build  communities,  and  has  an  influence  on  the  way  they  think  and  behave  through 
written  text -based  communication  (Bell,  2001;  Hine,  2000;  Markham,  1998;  Turkle,  1995).  Each  community  of 
practice  in  cyberspace  has  its  own  particular  language,  norms,  beliefs,  values,  and  local  histories.  Thus,  an 
ethnographic  qualitative  approach  was  useful  for  conducting  this  study  focusing  on  understanding  how  novice 
learners  become  part  of  the  community  of  practice  in  cyberspace. 

In  the  interpretive  paradigm,  people’s  actions  are  inherently  meaningful  and  are  understood  in  social  and 
cultural  contexts  that  constitute  the  action  (Schwandt,  2000).  Based  on  this  approach,  the  interpretive  paradigm 
researchers  select  a natural  place  where  the  participants  live  in  everyday  life  instead  of  experimental  settings  for 
understanding  how  people  make  a sense  of  their  life  toward  the  world.  In  the  natural  site,  researchers  play  a role  in 
interpreting  the  participants’  actions  embedded  in  social,  cultural,  economical,  historical,  and  political  aspects  as 
ethnography  researchers. 

Cyberspace  has  a great  number  of  natural  places  as  research  sites  for  an  interpretive  paradigm  inquiry.  Any 
websites  that  provide  dynamic  interactions  using  communication  tools  such  as  email,  discussion  boards,  chat  room, 
or  instant  messenger  without  any  control  to  people  who  go  to  the  websites  can  be  viewed  as  natural  sites.  Online 
communities  are  the  most  well-known  research  sites  for  ethnography  researchers  (Bayrn,  1995;  Denzin,  1999;  Hine, 
2000). 

Online  community  has  different  purposes  and  diverse  forms  using  different  communication  tools.  Despite 
these  differences,  online  communities  commonly  allow  people  to  post  their  opinion  or  idea,  to  read  others  message, 
and  to  reply  to  them  without  any  limit  of  time  and  place.  Some  people  may  only  read  others’  postings  as  lurkers 
without  any  response.  Other  people  may  go  to  an  online  community  irregularly  or  stop  going  there  suddenly.  In 
general,  online  communities  are  open  to  anyone  from  the  world  and  cannot  control  people  and  their  message.  In 
addition,  while  they  are  involved  in  these  online  communities,  people  think,  feel,  change,  and  learn  naturally 
(Markham,  1998;  Turkle,  1995).  Thus,  it  can  be  said  that  all  activities  that  occur  in  online  communities  are  natural. 
Whereas  many  studies  on  online  communities  have  been  conducted,  there  are  few  studies  on  online  communities 
related  with  education,  especially  schooling.  Like  other  online  communities,  online  communities  for  schooling  can 
have  common  ties  such  as  academic  topics  or  subjects  and  social  interactions  between  teachers  (experts)  and 
learners  (novice  learners). 

Research  Site 

The  research  site  of  this  study  was  a virtual  community  school  that  is  a non-profit  public  school  for 
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students  in  a mid-west  state.  This  school  offers  a statewide,  comprehensive  educational  program  for  students  in 
grades  K-12,  is  established  under  The  Charter  School  Law  of  the  state,  and  is  governed  by  the  School  Board  of  the 
state.  This  school  runs  all  courses  on  cyberspace  during  a whole  academic  year.  Students  are  provided  a computer 
and  all  kinds  of  technologies  they  need  for  learning  such  as  a scanner,  a printer,  the  Internet  service,  etc.  Students 
are  required  to  take  an  introductory  computer  literacy  class  before  their  classes  start.  After  classes  begin,  students 
will  have  access  to  the  virtual  classroom,  the  learning  website,  for  twenty  four  hours  per  day  and  seven  days  per 
week.  In  order  to  access  the  website,  they  need  a user  ID  and  password.  This  website  allows  only  students  and 
teachers  at  this  school  to  access  it. 

The  learning  website  was  created  specifically  for  this  school  by  a business  company,  instead  of  using  a 
platform  such  as  Blackboard  or  WebCT.  Each  subject,  regardless  of  grade,  has  a consistent  structure  in  the  website 
and  has  the  same  items:  Teachers’  Homepage,  Meet  the  teacher.  Class  help.  Timeline,  Syllabus,  Get  started,  Go  to 
school.  Email,  Forum,  Chat,  Office  Hours.  Students  can  use  email  or  telephone  for  communicating  with  their 
teachers.  Forum  enables  students  to  communicate  with  their  teachers  and  other  students  for  asking  a question  about 
their  learning  or  assignments  and  to  post  messages  related  with  content  area  and  reply  to  other  students  ’ messages 
for  active  discussion.  In  addition,  teachers  provide  students  with  synchronous  communication  tools,  chat  rooms  and 
Elluminate,  once  a week.  Any  students  who  have  some  questions  about  their  learning  or  assignments  can  attend 
teachers’  chat  session.  They  can  leave  chat  rooms  early  before  the  chat  session  is  over  if  they  get  answers  from  their 
teachers.  Elluminate  is  a virtual  classroom  where  teachers  and  students  gather  together  and  talk  to  each  other  by 
using  a headset  with  a microphone  in  cyberspace.  This  school  started  using  Elluminate  in  every  subject  of  the 
middle  school  at  the  second  semester  of  2003-2004.  1 conducted  this  study  during  the  entire  second  semester  of 

2003-2004  in  this  school. 

Participants 

In  the  interpretive  paradigm  focusing  on  an  in-depth  understanding,  an  ethnographic  researcher  relies  on 
small  samples  in  order  to  learn  a great  number  of  issues  of  central  importance  to  the  purpose  of  the  research.  In  this 
light,  unlike  quantitative  research,  selecting  samples  in  qualitative  research  tends  to  be  purposive  rather  than  random 
(Patton,  1990).  Thus,  the  participant  selection  for  my  study  was  based  on  a purposeful  sampling. 

The  participants  were  a Language  Arts  (LA)  teacher  and  seventeen  students  in  £h  grade  at  the  virtual 
community  school.  For  recruitment  of  the  teacher,  a school  director  at  this  school  made  a connection  between  the 
teacher  who  was  willing  to  participate  in  this  study  voluntarily  and  me . Since  teachers  at  this  school  worked  at  their 
home  around  the  state,  I could  not  meet  any  teachers  at  the  school  building  when  I met  the  director.  I was  informed 
about  the  teacher  by  email  a few  days  later  and  then,  sent  her  the  first  email  introducing  myself  and  this  study.  She 
replied  to  me  right  away  and  also  introduced  herself  a little  bit.  She  has  eighteen  years  of  teaching  experience  in 
physical  schools  and  it  was  her  first  year  to  work  at  this  school.  Even  though  we  did  not  meet  each  other  face-to- 
face  at  that  moment,  I felt  that  she  was  comfortable  with  me  and  excited  about  being  part  of  this  study.  I had  a few 
chances  to  meet  her  face-to-face  during  this  study  but  in  general,  she  and  I met  in  cyberspace  by  email. 

For  recruitment  of  students,  I asked  the  teacher  to  announce  this  study  on  her  homepage  where  every  LA 
student  visited  for  their  learning.  A few  days  later,  she  emailed  me  that  three  girls  out  of  around  ninety  students 
wanted  to  work  with  me  and  gave  me  their  names  for  me  to  contact  them.  In  order  to  recruit  more  students,  I 
decided  to  send  each  8th  grade  student  an  email  introducing  myself  and  this  study.  I also  attached  a recruitment 
letter  for  parents  to  each  email.  It  took  a few  hours  to  finish  sending  all  8th  grade  students  an  email.  Surprisingly,  I 
had  several  emails  from  some  students  who  were  interested  in  being  part  of  this  study  while  I was  sending  emails  to 
every  student.  Finally,  I had  eight  more  students,  six  girls  and  two  boys,  before  the  second  semester  started. 
However,  I lost  a girl  a few  weeks  later  after  the  second  semester  began  because  I could  not  get  any  emails  from  her. 
Just  in  case  to  lost  more  students  as  time  went  by,  I asked  the  teacher  to  keep  announcing  this  study  on  her 
homepage.  As  the  second  semester  went  by,  I had  seven  more  students  at  different  times.  Some  of  them  started 
working  with  me  at  the  beginning  of  the  second  semester  and  other  students  joined  this  study  in  the  middle  of  the 
second  semester.  The  reason  why  I decided  to  work  with  them  was  because  I thought  that  each  student  had  his/her 
own  valuable  experiences  of  going  to  this  school  and  then,  their  experiences  could  contribute  to  this  study. 

All  of  the  participants  participated  in  this  research  voluntarily.  As  for  informed  consent,  I sent  each 
participant  (and  parents)  an  informed  consent  form  by  mail  and  received  all  informed  consents  by  mail. 

In  order  to  create  rapport  with  my  participants,  I sent  them  emails  right  away  after  they  had  sent  me  an 
email  telling  me  that  they  wanted  to  be  part  of  this  study.  All  of  my  participants  were  willing  to  introduce 
themselves  to  me  and  felt  comfortable  with  me  from  the  beginning  of  our  communications  even  though  we  had 
never  meet  together  face-to-face  before.  I did  not  have  any  chances  to  meet  any  of  my  participants  during  my  data 
collection. 
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Data  Collection  Procedure 

Data  was  collected  by  Interviews,  participant  observation,  and  document  analysis.  These  are  described  in 
detail  in  the  following  section. 

Interview 

For  ethnographic  research  based  on  the  interpretive  paradigm,  interviews  are  “the  most  common  and 
powerful  ways”  (Fontana  & Frey,  2000,  p.  645)  in  which  we  can  understand  other  people.  Interviews  attempt  to 
understand  the  world  from  the  participants’  points  of  view,  to  unfold  the  meaning  of  people’s  experiences,  and  to 
uncover  their  lived  world  (Kvale,  1996).  Kvale  (1996)  states  that  interviews  have  strength  in  that  researchers  can 
see  participants’  inner  view  instead  of  simply  getting  answers  to  their  questions.  Based  on  its  characteristics, 
interviews  are  selected  as  a main  research  method  for  data  collection  in  cyberspace  (Correll,  1995;  Markham,  1998). 

Researchers  can  use  email  or  chat  rooms  for  having  an  interview.  Email  has  strength  in  that  participants 
can  take  enough  time  to  answer  to  interviews  questions  and  then,  can  provide  a researcher  with  rich  data  (Correll, 
1995;  Ferri,  2000;  Hine,  2000).  However,  it  lacks  in  the  flexibility  of  an  “in  person”  interview  because  participants 
tend  to  answer  all  questions  without  communicating  with  a researcher.  Chat  rooms  provide  the  flexibility  of  face-to- 
face  interviews  in  cyberspace  in  that  it  is  conducted  synchronously  in  real  time  (O'Connor  & Madge,  2000,  cited  in 
Mann  & Stewart,  2000).  In  this  light,  I used  chat  for  conducting  interviews  with  my  participants. 

The  interviews  with  my  participants  were  conducted  once  a week.  I met  each  participant  at  a chat  room  of 
Language  Arts  class  for  an  hour.  In  order  to  make  sure  all  of  my  participants  felt  comfortable  with  having  an 
interview  at  this  chat  room,  I had  two  informal  chats  with  each  of  them  before  the  first  interview.  My  participants 
and  I used  email  the  virtual  community  school  provided  to  make  an  appointment  for  an  interview  or  to  change  their 
interview  schedule.  At  the  beginning  of  the  first  interview,  I told  my  participants  some  instructions  they  needed  to 
remember  for  having  an  interview.  One  of  the  instructions  I gave  them  was  to  write  down  “done”  or  “next”. 
Because  I could  not  see  them  face-to-face,  it  was  hard  for  me  to  know  whether  they  finished  telling  me  their  stories 
to  an  interview  question.  Another  important  instruction  was  that  they  did  not  have  to  answer  some  questions  that 
made  them  feel  uncomfortable.  Fortunately,  there  were  no  interview  questions  that  made  them  feel  uncomfortable. 
In  general,  my  participants  were  pretty  open-minded  and  willing  to  talk  to  me  about  their  experiences  at  the  school 

In  addition  to  one-on-one  interviews,  I conducted  group  interviews  for  three  weeks.  Since  my  participants 
were  diverse  in  terms  of  gender,  race,  and  previous  schools  background  before  going  to  the  virtual  community 
school,  I decided  to  have  three  group  interviews  for  three  weeks.  I made  each  group  have  three  or  four  participants 
and  let  them  talk  to  each  other  about  some  broad  interview  questions.  It  took  around  one  hour  or  one  and  half  hours. 
From  these  group  interviews,  I found  that  my  participants  were  so  excited  about  talking  to  other  students  and  got  to 
know  each  other  quickly  in  cyberspace  even  though  they  had  never  met  together  face-to-face.  After  three  group 
interviews,  I resumed  one-on-one  interviews  for  the  last  two  weeks. 

Participant  Observation 

Participant  observation  is  considered  as  central  to  most  of  ethnographic  research  (Wolcott,  1999).  While 
observation  allows  researchers  to  see  the  participants’  actions  objectively  as  the  stance  of  the  “fly  on  the  wall” 
(Roman,  1993,  cited  in  Proweller  (1998)),  participant  observation  makes  researchers  involve  in  active  interactions 
such  as  participation  in  classroom  discussion  or  group  activities  with  participants  within  the  research  site.  This 
participant  observation  provides  researchers  with  better  understanding  the  context. 

It  is  participant  observation  that  many  researchers  strongly  suggest  for  an  ethnographic  research  in 
cyberspace  (Mann  & Stewart,  2000).  What  researchers  can  observe  in  cyberspace  is  not  participants  ’ visible  actions 
but  written  texts  they  write  in  email  or  chat  roomor  on  discussion  boards.  Since  people  cannot  have  face-to-face 
interactions  in  cyberspace,  all  interactions,  in  general,  depend  on  written  texts.  Even  though  we  need  to  admit  that 
cyberspace  provides  more  than  text,  it  can  be  said  that  written  text  is  a unique  form  of  interaction  in  terms  of  online 
communication  (Mitra  & Cohen,  1999). 

I conducted  both  observation  and  participant  observation.  For  observation,  I observed  the  school 
announcement  that  was  the  first  webpage  after  students  logged  in  in  daily  basis  in  order  to  understand  what  was 
going  on  at  the  school.  I also  observed  the  Language  Arts  teachers’  homepage  in  daily  basis.  Observing  the  LA 
teacher’s  homepage  helped  me  understand  what  was  going  on  in  her  classroom  and  how  she  communicated  with  her 
students  in  cyberspace. 

I observed  chat  rooms  of  Language  Arts  once  a week.  Since  the  LA  teacher  used  chat  rooms  for  students 
asking  a question  about  their  learning  content  or  assignments/projects  instead  of  discussion,  participant  observation 
was  not  needed.  The  LA  teacher  used  chat  room  for  communicating  with  her  students  until  in  the  middle  of  the 


890 


second  semester.  After  the  middle  of  the  second  semester,  she  used  only  Elluminate. 

I also  observed  Ellumiate  of  Language  Arts  once  a week  from  the  beginning  of  the  second  semester. 
Unlike  chat  rooms,  Elluminate  allowed  only  nine  students  to  attend  at  a time.  Thus,  the  LA  teacher  announced  the 
day  and  time  of  Elluminate  at  the  first  day  of  the  week  on  her  homepage  and  encouraged  students  to  attend  it.  In 
order  to  attend  Elluminate,  students  needed  to  have  ID  and  password  and  needed  to  have  new  ones  for  every  time. 
Since  there  were  not  many  students  who  attended  Elluminate  every  week,  I was  able  to  observe  every  Elluminate 
session  during  the  whole  second  semester.  At  the  beginning  of  the  each  Ellunimate  session,  the  LA  teacher  asked 
students  to  introduce  themselves  to  the  class  since  they  could  not  see  each  other  face-to-face.  All  students  and  the 
teacher  could  see  in  the  Elluminate  was  their  ID  instead  of  real  name.  In  Elluminate,  the  LA  teacher  encouraged 
students  to  be  more  engaged  in  their  learning.  When  she  explained  part  of  the  learning  content,  for  instance,  she 
wanted  to  check  if  students  understood  her  well.  For  doing  this,  she  frequently  asked  students  if  they  had  any 
questions  so  far  while  explaining.  Each  student  answered  to  her  whether  they  had  a question  or  not  by  speaking  to 
her  directly  or  by  clicking  on  “Smile”  button.  In  addition,  students  were  allowed  to  click  on  “Raise  hand”  button 
whenever  they  had  a question.  Even  though  Elluminate  was  a virtual  classroom  in  cyberspace,  interactions  between 
the  teacher  and  students  were  active  and  dynamic.  While  I observed  Elluminate,  I wrote  field  notes  as  ethnographic 
qualitative  researchers  in  physical  spaces  do. 

For  participant  observation,  I observed  Forum  of  Language  Arts  and  participated  in  it.  Around  a month 
after  the  second  semester  started,  the  LA  teacher  wanted  to  encourage  students  to  be  involved  in  discussion  on  some 
learning  contents  in  Forum.  Since  she  used  Forum  for  the  firs  time  at  the  school,  she  wanted  me  to  participate  in 
making  some  discussion  questions  and  replying  to  students’  postings.  I read  the  learning  contents  for  discussion  and 
read  and  replied  to  all  of  students’  postings  in  Forum. 

I also  conducted  participant  observation  by  reading  some  students’  projects.  A couple  of  students  wanted 
me  to  take  a look  at  their  projects  and  to  give  them  some  feedback  on  it  before  they  submitted  them.  They  sent  me 
their  projects  as  an  attached  file  by  email  and  then,  they  and  I exchanged  our  opinions  on  them  by  email. 

Document  Analysis 

In  an  ethnographic  qualitative  approach,  documents  analysis  is  important  as  much  as  interviews  and 
observation  in  that  documents  reflect  historical  situations  as  a stable  source  and  are  embedded  in  individual’s  social 
and  cultural  contexts  in  everyday  life  (Hodder,  2000;  Lincoln  & Guba,  1985).  While  some  researchers  distinguish 
documents  and  records  on  the  basis  of  whether  the  texts  are  personal  or  private  (Lincoln  & Guba,  1985),  any  kinds 
of  documents  can  be  counted  for  document  analysis  according  to  research  purposes.  In  this  respect,  documents 
include  public  records  such  as  government  reports  or  media  accounts,  private  documents  such  as  diaries,  reflection 
journal,  or  letters,  books,  photographs,  video,  etc.  (Schwandt,  2001). 

With  the  advent  of  the  Internet,  cyberspace  produces  another  kinds  of  documents  that  have  never  existed 
before  in  physical  spaces.  Web  pages,  emails,  postings  on  discussion  boards,  and  messages  on  chat  rooms  or  instant 
messengers  are  related  with  people ’s  day-to-day  lives.  Thus,  researchers  who  conduct  an  ethnographic  qualitative 
inquiry  in  cyberspace  may  need  to  analyze  some  of  these  new  kinds  of  documents  for  an  in-depth  understanding  of 
participants’  actions. 

For  this  study,  web  pages  of  school  announcements,  web  pages  of  the  LA  teachers’  announcements, 
learning  content  of  Language  Arts,  emails  my  participants  exchanged  with  each  other,  and  messages  my  participants 
posted  in  Forum  were  collected  for  document  analysis.  Since  all  of  these  documents  were  on  the  Internet,  I did  not 
have  to  print  them  all.  Instead,  I was  able  to  copy  and  paste  hundreds  of  emails  and  many  postings  in  Forum  into  a 
Microsoft  Word  file.  For  collecting  the  two  kinds  of  web  pages,  I saved  them  as  image  files  using  “PrintScreen”  on 
the  keyboard.  I printed  out  only  learning  content  of  Language  Arts  because  it  was  hard  to  copy  and  paste  them. 

Preliminary  Analysis  of  Data 

In  this  section,  I briefly  present  three  emerging  findings  I have  found  and  these  findings  will  be  elaborated 
in  further  analysis. 

Perception  of  Online  School 

Some  of  my  participants  seemed  to  think  that  the  reason  for  going  to  their  online  school  was  to  have 
flexibility  of  studying.  A boy  told  me  that  it  was  easier  to  learn  at  his  own  space  in  his  online  school.  A girl  told  me 
that  she  liked  that  she  could  make  her  own  hours  to  study.  Another  boy  said  that  he  got  to  do  his  work  at  his  own 
space  and  anytime  during  day  pretty  much.  For  this  boy,  he  liked  to  study  during  afternoon  or  night  instead  of 
morning.  But,  there  was  a negative  case  of  this.  A girl  told  me  that  she  could  go  more  at  her  own  speed  in 
home  shook 
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Other  my  participants  seemed  to  think  that  going  to  their  online  school  would  make  them  enjoy  going  to 
school.  Several  students  told  me  that  it  was  pretty  cool  to  work  on  the  computer  and  a kind  of  fun  because  they 
liked  being  on  the  Internet  itself  and  because  schoolwork  was  a little  easier  than  public  schools  or  homeschool. 

A few  students  seem  to  think  that  going  to  their  online  school  was  to  go  to  an  actual  school.  A girl  said  to 
me  “I  mean  we  have  homework  that  is  due  at  a certain  time  and  if  it  isn't  done  than  there  are  consequences.  WE 
have  teachers,  who  grade  your  papers  and  correct  what  you  need  to  be  corrected.  We  have  administration.  A 
principal.”  Another  girl  also  told  me  that  she  felt  going  to  “a  school”  because  she  had  an  actual  teachers. 

Learning  in  Cyberspace:  Email  as  a Communication  Tool 

Even  though  the  school  email  was  a main  communication  tool  between  teachers  and  my  participants,  all  of 
my  participants  told  me  that  no  one  at  the  school  taught  them  how  to  use  it.  Thus,  it  was  interesting  to  understand 
both  how  my  participants  learned  how  to  use  the  school  email  and  how  they  practiced  for  fully  participation  in 
communication  with  their  teachers  and  other  students  in  the  school.  Some  of  my  participants  who  already 
experienced  using  regular  email  seemed  to  feel  comfortable  with  the  school  email  at  first.  They  were  a little  familiar 
with  a basic  structure  of  email.  Based  on  their  previous  experiences  of  a regular  email,  they  tried  clicking  on  every 
button  in  the  school  email  system.  They  had  some  problems  of  sending  an  email  or  replying  to  an  email  at  the 
beginning  of  their  trial.  In  this  case,  some  of  my  participants  asked  their  parents  to  help  them  out.  One  boy  told  me 
that  his  neighbor  showed  him  how  to  use  the  school  email  and  that  he  observed  what  she  had  done. 

Other  participants  who  had  never  used  a regular  email  before  their  online  school  had  a little  different 
experience  of  learning  how  to  use  the  school  email.  They  seemed  to  try  to  find  some  information  from  the  online 
school  first.  One  girl  told  me  that  she  took  a look  at  a section  ‘‘getting  started”  at  one  of  the  teachers’  homepage 
first.  She  read  all  of  the  information  about  using  email  in  the  section  and  then,  tried  to  follow  the  direction.  Even 
though  she  needed  to  practice  for  using  the  school  email  for  a while,  she  was  satisfied  with  the  way  the  school 
provided  information. 


Teachers/Students  Power  Disparity 

Almost  all  my  participants  tended  to  think  that  teachers  at  the  school  were  nicer,  understanding,  and 
friendly.  They  seemed  to  think  that  the  relationship  between  their  teachers  and  themselves  were  more  equal,  in 
general.  A girl  told  me  that  she  was  very  surprised  when  one  of  her  teachers  always  replied  to  her  and  talked  to  her 
about  her  daughter  once.  She  said  to  me  that  she  was  very  happy  because  the  teacher  really  “talked”  to  her. 
However,  I also  found  different  interaction  between  teachers  and  students  in  communicating  by  email.  A girl  told 
me  that  she  was  frustrated  and  finally  stopped  emailing  one  of  her  teachers  when  she  argued  with  the  teacher  about 
some  wrong  answers  from  quiz.  Since  this  was  a preliminary  analysis  on  power  disparity  between  teachers  and 
students,  I need  to  analyze  this  in  detail. 


Further  Analysis 

With  an  elaboration  of  above  findings,  I am  going  to  analyze  data  on  online  identity  of  students,  online 
school  activities,  novice  students’  practice  in  online  school  community,  and  discourse,  subjectivity,  and  agency  of 
students  and  teachers  in  online  school  community. 

'This  paper  is  based  on  part  of  the  author’s  dissertation  research  at  The  Ohio  State  University,  Suzanne  Damarin, 
advis  or. 

2In  this  paper,  I use  the  term  physical  space  instead  of  “real  environments”  or  “reality”  because  these  two  words  tend 
to  signigy  dualism  - physical  space  is  real  and  cyberspace  is  not  real. 
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Introduction 

Despite  the  fact  that  creativity  has  long  been  a topic  of  interest  to  educators,  artists,  and  scientists,  it  has  not 
been  a subject  of  serious  study  among  experimental  psychologists  (Finke,  Ward,  & Smith,  1992).  This  lack  of 
scientific  studies  on  creativity  appears  to  relate  to  the  difficulty  of  conceptualizing  or  defining  exactly  what 
creativity  or  creative  behavior  is.  While  there  are  many  definitions  of  creativity,  no  one  definition  is  universally 
accepted  (Treffinger,  Young,  Selby,  & Shepardson,  2002). 

Creativity  Definitions 

In  early  research,  creative  ability  was  not  separated  from  traditional  intelligence;  often  they  were  treated  as 
one  capacity  or  ability  (Runco,  2002).  A more  recent  trend  has  resulted  in  a somewhat  extreme  contrast  between 
creativity  and  intelligence.  For  example,  a general  view  developed  that  considered  creativity  to  involve  divergent 
thinking  while  traditional  intelligence  involves  convergent  thinking.  More  recently,  an  integrated  view  has  evolved 
that  views  creativity  as  a process  which  involves  the  combination  of  both  divergent  and  convergent  thinking  (Runco, 
2002). 

According  to  Runco  (2002)  studies  of  creativity  are  typically  categorized  in  terms  of  personality,  product, 
process,  and  press.  Personality  studies  generally  identify  the  personal  characteristics  that  distinguish  creative 
persons  from  less  creative  persons.  Creative  personality  has  been  suggested  to  include  traits  such  as  courage, 
openness,  curiosity,  autonomy,  playfulness,  risk-taking,  tolerance  for  ambiguity  and  so  forth.  In  contrast,  studies  of 
creative  products  examine  artifacts  such  as  works  of  art,  or  publications.  Such  products  are  evaluated  on  the  basis  of 
preset  criteria  involving  factors  such  as  novelty  and  usability.  Studies  that  focus  on  process  examine  the  underlying 
progression  or  development  of  creative  performances  or  products.  Runco  used  the  term  press  to  describe  the  studies 
that  examine  the  interaction  between  creative  behaviors  and  environments. 

Among  these  varied  approaches  to  studying  creativity,  the  process  approach  is  considered  here  to  provide  a 
more  functional,  integrated  view  of  creativity.  Process  includes  many  aspects  of  the  other  approaches.  The 
processes  involved  while  examining  creativity  performance  include  components  such  as  the  individual’s  sequence  of 
responses,  spatial  elements  of  product  development,  temporal  elements  of  product  development,  underlying  thinking 
processes,  and  final  products.  These  process  factors  will  provide  a means  of  investigating  differences  in  response 
patterns  among  individuals  to  provide  a more  complete  understanding  of  the  nature  of  creativity.  Further,  the  study 
of  the  process  approach  may  help  to  explain  the  ways  divergent  and  convergent  thinking  operate  to  complete  the 
creative  performances. 

A creative  performance  involves  a dynamic  process  including  interaction  with  problems,  prior  knowledge, 
and  social  contexts.  However,  when  reviewing  the  literature,  it  is  apparent  that  there  is  a lack  of  research  or  sound 
methodologies  for  examining  the  dynamic  process  components  of  creativity.  At  minimum,  there  is  mismatch 
between  proposed  definitions  and  methods  used  to  investigate  them.  Furthermore,  current  instruments  used  to 
assess  creativity  (e.g.  Torrance  Tests  of  Creative  Thinking  (1974)  have  focused  primarily  on  examining  final 
products  and  have  been  limited  in  providing  information  about  the  underlying  processes  of  creative  production. 

Measuring  Process  in  Creative  Productions 

The  development  of  new  methodologies  or  techniques  that  provide  for  the  measurement  of  the  process 
elements  associated  in  a creative  performance  may  be  instrumental  in  furthering  our  understanding  of  the  nature  of 
creativity.  In  fact,  the  introduction  of  such  methodologies  may  not  only  provide  a better  picture  of  creativity  but 
may  allow  for  new  functional  definitions  of  creativity  itself.  Similarly,  such  development  may  apply  to  monitoring 
processes  or  thinking  patterns  of  participants  who  are  engaged  in  creative  problem  solving  activities.  In  the  past  it 
has  been***  to  examine  the  mental  processes  a person  goes  through  while  endeavoring  to  solve  a problem 
creatively,  resulting  in  mostly  relying  on  anecdotal  or  introspective  accounts.  Current  computer  technologies  have 
produced  environments  and  resources  that  offer  much  promise  to  research  on  creativity  and  it's  underlying 
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processes.  Not  only  can  we  develop  more  efficient  and  convenient  ways  to  gather,  store  and  distribute  creative 
products  for  evaluation,  but  we  can  now  gather  a vast  amount  of  information  about  the  processes  used  in  producing 
those  products.  Computer  technology  now  makes  it  possible  to  provide  unobtrusive,  automatic  recording  of 
multiple,  concurrent  components  of  examinee  performance.  The  purpose  of  this  presentation  is  to  illustrate 
computerized  methodologies  that  monitor  and  record  process  components  such  as  response  patterns,  temporal 
elements,  and  sequential  stages  of  creative  products.  In  addition,  local  and  internet-based  resources  will  be 
discussed  in  relation  to  storage  of  data  and  products,  as  well  as  efficient  distribution  for  analysis  and  evaluation. 

Defining  Creativity  as  a Thinking  Process 

Creative  performances  require  a person  to  be  involved  in  a task  for  a while  and  to  execute  his  individual 
resources  to  complete  it.  In  dealing  with  a task  demanding  creative  ability,  individuals  may  utilize  different  types  of 
thinking  processes  to  respond  or  adapt  to  the  constraints  that  the  task  presents.  The  underlying  thinking  patterns 
required  for  creative  performance  should  be  studied  in  relation  to  their  similarities  to,  and  differences  from  those 
needed  for  other  types  of  cognitive  performance.  A number  of  approaches  to  the  study  of  creativity  and  related 
processes  have  emerged  over  the  years. 

According  to  Graham  Wallas  (1926)  the  creative  problem  solving  process  goes  through  a sequence  of 
stages:  preparation,  incubation,  illumination,  and  verification.  The  preparation  stage  involves  the  process  of 
investigating  a problem  in  all  directions  and  accumulating  knowledge.  Incubation  is  the  process  of  consciously 
focusing  on  problems  other  than  the  proposed  one.  It  is  “voluntary  abstention  from  conscious  thought  on  any 
particular  problem.”  (p.86).  The  illumination  stage  involves  the  moment  when  an  inspiration  emerges  as  the  result 
of  unconscious  work.  At  the  verification  stage  the  validity  of  the  idea  is  tested. 

In  studying  creativity  as  a process,  some  researchers  have  attempted  to  explain  creativity  in  the  same 
context  as  problem  solving  processes.  Torrance  (1974)  defined  creativity  as  a process  for  a person  to  endeavor  to 
find  a solution  to  a problem.  Accordingly,  creative  thinking  is  a process  in  which  individuals  sense  problems,  form 
hypotheses,  communicate  the  results,  and  modify  and  embellish  the  hypotheses.  Based  on  this  definition,  Torrance 
created  a divergent  thinking  test,  the  Torrance  Tests  of  Creative  Thinking  (date),  which  focused  on  measurement  of 
originality,  fluency,  flexibility,  and  elaboration. 

Guilford  ( 1950)  was  also  interested  in  the  types  of  thinking  involved  in  creative  performances.  He 
proposed  two  general  types  of  thinking,  divergent  thinking  and  convergent  thinking.  In  convergent  thinking,  the 
individual  is  expected  to  ‘converge’  on  the  appropriate  answer  to  a problem,  while  divergent  thinking  involves 
producing  a range  of  novel  solutions  to  a problem  (Robertson,  1999).  Treffinger  et  al  (2002)  reviewed  Guilford's 
approach  and  emphasized  his  view  that  problem  solving  and  creative  thinking  are  closely  related  and  his  position 
that  the  definitions  for  the  two  types  of  thinking  have  a logical  connection.  “Creative  thinking  produces  novel 
outcomes,  and  problem  solving  involves  producing  a new  response  to  a new  situation,  which  is  a novel  outcome” 
(Guildford,  1977,  p.  161 , cited  by  Treffinger  et  al,  2002). 

Similarly,  Treffinger  et  al.  (2002)  defined  creativity  from  the  combined  perspective  of  convergent  and 
divergent  thinking.  They  substitute  ‘generating  ideas’ for  divergent  thinking.  Generating  ideas  includes  fluency, 
flexibility,  originality,  elaboration  and  metaphorical  thinking.  Convergent  thinking,  in  their  view,  is  replaced  by 
‘digging  deeper  into  ideas’  which  includes  analyzing,  synthesizing,  reorganizing  or  redefining,  evaluating,  seeing 
relationships,  desiring  to  resolve  ambiguity  or  brining  order  to  disorder,  and  preferring  complexity  or  understanding 
complexity. 

Ebert  (1994)  also  viewed  creative  thinking  to  include  cognitive,  convergent  thinking.  In  his  cognitive 
spiral  model  he  argued  a conceptualized  creative  thinking  is  an  integral  component  of  all  cognitive  processing.  The 
spiral  model  adopted  the  sequential  nature  of  the  information-processing  model  by  delineating  five  components: 
perceptual  thought,  creative  thought,  inventive  thought,  metacognitive  thought,  and  performance  thought.  Ebert 
arranged  these  components  in  a spiral  continuum  rather  than  the  cycle  typical  of  the  information-processing  model. 
The  dynamic  aspect  of  a creative  mind  interrelating  prior  knowledge  or  stimuli  is  an  important  component. 

Amabile's  (1983)  definition  provided  a broader  view  for  creative  thinking  by  including  an  interaction 
component  between  individuals  and  their  environments.  He  argued  that  creativity  is 

best  conceptualized  not  as  a personality  trait  or  a general  ability  but  as  a behavior  resulting 
from  particular  constellations  of  personal  characteristics,  cognitive  abilities,  and  social 
environments.  This  behavior,  which  is  evidenced  in  products  or  responses,  can  only  be 
completely  explained  by  a model  that  encompasses  all  three  sets  of  factors,  (p.358). 

Similarly,  Treffinger  et  al.  (2002)  argued  that  creative  productivity  arises  from  the  dynamic  interactions 
among  four  essential  components:  personal  characteristics,  operations,  context,  and  outcomes.  According  to  the 
authors,  operations  are  the  strategies  and  techniques  that  people  employ  to  generate  and  analyze  ideas,  solve 
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problems,  make  decisions,  and  manage  their  thinking. 

In  summarizing,  there  is  no  universal  definition  of  creativity  and,  consequently,  no  agreement  on  the  best 
way  to  measure  it.  Creativity  is  described  here  as  a dynamic  thinking  process  that  interacts  with  problems,  prior 
knowledge,  and  social  contexts.  Creativity  as  a dynamic  process  entails  both  divergent  and  convergent  thinking 
processes.  The  components  associated  with  creativity  and  creativity  research  are  grouped  into  the  following 
categories:  Personality,  Press,  Process  and  Product.  Process  and  product  are  the  major  elements  of  any  creativity 
production  and  are  mediated  or  influenced  by  personality  and  press  factors.  All  four  work  in  combination  and  have 
influence  over  each  other  as  the  individual  develops  and  matures  over  time. 


Personality 


Creative  Production 
Process  ^^^^^^^P1Qduct 


Press  (Environment) 


Figure  1 . Creative  production  consists  of  a process  and  a resulting  product  which  are  influenced  by  past  and  present 
personality  and  environment  components. 

The  measurement  techniques  presented  here  will  focus  on  the  process  and  product  components  of  any 
creative  production.  Some  elements  of  personality  and  press  components  (especially  the  testing  setting)  will  be 
addressed  where  appropriate. 


Measuring  creative  thinking 

When  measuring  creativity,  focus  in  the  past  has  often  been  only  placed  on  examining  creative  end 
products.  However,  when  creativity  is  defined  and  measured  in  this  manner,  the  observation  of  the  dynamic  aspect 
of  the  creative  process  is  missed.  Even  though  some  efforts  to  illuminate  the  creative  process  have  been  made,  most 
of  the  studies  have  focused  on  anecdotal  and  introspective  accounts  (Finke  et  ah,  1992;  Kwon,  1996).  This 
informal,  descriptive  approach  to  the  construct  of  creativity  makes  it  difficult  to  study  creativity  within  the  context 
of  controlled  scientific  experiments  (Finke  et  ah,  1992).  More  objective  measurement  techniques  are  necessary  for  a 
complete  analysis  of  creative  abilities. 

In  addition,  most  of  the  attempts  to  assess  the  creativity  used  pencil-paper  tests  that  add  little  to  the  ability 
to  observe  the  underlying  process.  For  example,  the  Torrance  Tests  of  Creativity  Thinking  (1974),  one  of  the  most 
widely  used  creativity  tests  in  paper-pencil  format  (Cooper,  1991),  has  been  used  extensively  to  examine  the 
creative  thinking  process.  The  research  on  creativity  using  this  test  has  therefore  focused  on  the  test's  scoring  of 
final  outcomes.  As  Erdos  argued  (1990),  this  approach  would  yield  little  information  about  the  processes  involved 
in  creative  thinking  and  it  is  necessary  to  move  to  a more  dynamic  approach. 

Developing  an  instrument  to  support  a dynamic  approach  for  creativity  is  essential  because  the  derived 
information  through  such  instruments  can  reveal  a more  complete  understanding  of  creativity.  The  data  obtained  by 
using  tools  which  provide  observation  of  process  will  bring  us  more  in-depth  information  about  how  individuals  go 
about  creative  performances,  what  their  patterns  of  creative  behaviors  are,  and,  consequently,  what  types  of  thinking 
accompanies  creative  behaviors.  In  addition,  the  increased  understanding  obtained  by  employing  such  tools  will 
help  our  conceptual  reframing  of  the  construct  of  creativity.  Given  the  current  situation  in  which  researchers  have 
no  universal  definition  of  creativity,  such  tools  should  be  of  great  benefit  in  comparing,  refining,  and  combining  the 
competing  theoretical  views.  While  it  is  typically  not  desirable  to  create  theories  based  primarily  on  the  tools 
available  (e.g.  everything  becomes  a nail  when  you  have  a hammer),  the  reliance  on  advances  in  measurement  and 
observation  technologies  does  provide  a legitimate  influence  for  expanding  the  definition  of  creativity.  We  are  no 
longer  bound  by  paper  and  pencil  resources  and  the  consequent  distribution  of  only  final  products  for  evaluators  to 
score.  We  can  now  take  advantage  of  computer  technologies  that  make  it  possible  to  monitor  and  record  a wide 
range  of  components  of  an  examinee’s  production  activities  and  make  them  available  for  research  or  testing 
purposes.. 

Proposed  creativity  measurement  method 

Based  on  the  above  views,  creativity  should  be  measured  not  only  by  final  products.  Consequently,  it 
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would  be  important  to  develop  methodologies  that  not  only  evaluate  creative  products,  but  also  enable  evaluation  of 
patterns  and  actions  used  to  generate  those  products.  Such  measurement  has  become  possible  using  computer- 
administered  testing  formats  that  provide  methods  to  readily  enable  the  tracking  and  recording  of  such  response 
patterns.  Computers  can  automate  monitoring  of  quantitative  variables  such  as:  response  latencies,  speed,  accuracy, 
numbers  of  attempts,  confidence  levels,  and  a record  of  self -correction  activities  (Johnson,  1982;  Kwon,  Goetz  & 
Zellner,  1998). 

This  presentation  will  demonstrate  a computer-based  methodology  that  automates  monitoring  and 
recording  of  examinees’  responses  while  they  are  involved  in  creative  activities.  Resources  and  techniques  have 
been  developed  that  use  both  local  and  internet  components  to  present  testing  materials,  monitor  both  overt  and 
underlying  responses,  and  then  store/manage  the  performance  data  and  final  products.  In  addition,  the  collected  data 
can  be  distributed  via  the  internet  to  evaluators  at  various  locations  or  placed  on  servers  for  analyses.  This 
distribution  capability  will  enhance  both  the  opportunities  for  creativity  research  as  well  as  the  accessibility  of  any 
resulting  tests. 

These  resources  will  have  multiple  advantages  over  traditional  paper-pencil  tests.  Overall  testing  would  be 
more  efficient  due  to  ease  of  replicating  and  accessing  the  testing  materials.  Cost  would  be  reduced  since  replication 
of  the  test  material  would  require  no  additional  physical  resources.  This  computer-based  method  would  allow  more 
careful  qualitative  observation  of  the  examinees  by  the  test  administrators  because  it  frees  them  from  other 
responsibilities  that  paper-pencil  testing  would  require,  e.g.  distributing/managing  testing  material  and  giving 
instructions.  All  timing  and  sequencing  of  events  and  management  of  associated  materials  would  be  under  the 
control  of  the  computer  and  would  serve  to  increase  standardization  of  the  administration  procedures.  For  example, 
such  functions  would  eliminate  the  need  for  examiners  using  stop  watches  to  monitor  particular  activities  thereby 
reducing  potential  social  pressure  as  an  influence  on  performance.  In  addition,  with  this  programmed  testing 
material,  it  is  possible  to  automate  keeping  track  of  quantitative  variables  such  as  response  latencies,  speed,  numbers 
of  attempts,  and  self-correction  activities.  This  method  also  enables  the  automatic,  unobtrusive  recording  of 
responses  on  a repeated  basis  giving  a more  complete  picture  of  examinee  performance.  For  example,  the  method 
proposed  here  was  programmed  to  record  the  stages  of  the  examinee’s  drawing  product  by  grabbing  snapshots  every 
10  seconds  providing  the  opportunity  to  analyze  temporal  and  spatial  development  patterns.  Temporal  patterns 
within  and  between  performance  tasks  can  also  be  measured  more  accurately.  In  addition,  other  tendencies  of 
creative  approaches  can  be  examined  through  the  recorded  sequence  of  drawings.  For  example,  some  responders 
might  prefer  to  move  from  whole  to  details,  while  others  tend  to  go  from  details  to  whole.  Some  might  keep  the 
original  idea  intact  through  the  whole  activity;  others  may  change  part  or  all  of  their  work  before  deciding  on  the 
final  product.  How  and  when  they  enter  or  edit  a title  for  their  products  can  also  be  recorded.  Parallel  data 
collection  could  be  introduced  such  as  embedded  means  for  the  examinees  to  rate  their  comfort  with  a task,  self- 
confidence  in  performance,  satisfaction  with  their  production,  etc.  These  ratings  would  also  provide  patterns  with 
respect  to  timing  and  change  patterns. 

Further,  this  method  is  advantageous  in  that  it  facilitates  the  process  of  delivering  the  collected  data  and 
analyses.  Automatically  recorded  data  will  be  delivered  via  Internet  and  arranged  as  database  files  in  individual 
profiles.  This  process  would  make  the  administration  and  analyses  more  efficient.  In  addition,  this  data  storage 
format  would  make  it  possible  for  evaluators  to  access  the  data  from  remote  locations  for  the  analyses  that  must  be 
done  by  human  observation.  Consequently,  in  exploring  the  behavior  patterns  of  creative  performances,  this  method 
should  increase  testing  validity.  Blind  raters  and  trained  experts,  who  do  not  know  about  the  study  hypotheses, 
subjects,  etc.,  could  be  selected  from  distant  places  and  asked  to  conduct  independent  analyses  of  the  data.  Having 
raters  from  different  settings  would  make  the  evaluation  of  drawings  more  objective  but  the  evaluations  could  still 
be  easily  acquired  and  managed. 

When  the  creativity  test  material  is  completed,  a pilot  test  of  the  proposed  test  material  will  be  conducted  to 
evaluate  the  presentation  resources,  interactivity,  examinees’  reactions  to  the  experience,  and  the  data  collection 
processes.  The  materials  will  be  refined  or  redeveloped  based  on  theses  findings.  The  intent  is  to  first  explore  the 
mechanics  of  measurement  and  evaluation  and  then  examine  the  data  to  determine  the  relative  value  of  the  types  of 
measurements  based  on  the  nature  of  the  patterns  observed.  Refinement  of  the  techniques  and  types  of  data 
collected  would  follow  according  to  the  initial  findings.  This  ongoing  evolution  of  measurement  techniques  and 
determination  of  appropriate  responses  to  observe  will  facilitate  our  understanding  of  the  relation  between  the 
creative  thinking  process  and  other  variables,  such  as  personality  traits,  motivation,  and  attitudes.  Such 
understanding  should  eventually  influence  our  definitions  of  creativity  and  lead  to  further  development. 

Examples:  Original  Testing  Format 

The  following  test  items  were  originally  developed  in  HyperCard  and  have  been  converted  in  Revolution,  a 
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development  environment  that  provides  additional  display  and  data  collection  components,  cross-platform  delivery. 


and  internet  transfer/accessibility. 


Rctiuity  I.  PICTURE  CONSTRUCTION 

Look  at  the  curved  shape  below  Hunk  of  a picture  or  an  object  wteh  you  can 
draw  with  the  shape  as  a part 

TVy  to  think  of  a pcture  trnt  no  one  else  will  think  of  Keep  adding  new  ideas  to 
your  first  idea  to  make  it  teU  as  interesting  and  as  exciting  a story  as  you  can 
when  you  have  completed  your  picture,  think  up  a name  or  title  up  a name  or  title 
for  it  Mid  wnte  it  at  the  bottom  of  the  page  m the  space  provided  Make  your  title 
jf — | as  clever  and  unusual  as  possible  Use  it  to  help  tell  yow  story 
f Toob  | 


Figure  2.  Picture  Construction-  Examinee  constructs 


drawing  ("think  of  a picture  or  an  object  which  you 
can  draw  with  this  shape  as  a part”)  and  adds  a title 
for  the  drawing.  All  time  and  choice  patterns  are 
recorded.  The  instructions  are  removed  during  the 
drawing  on  all  tasks . 


Pencil 


-IS 


Move 


By  ad±ng  lines  to  the  incomplete  figures  on  the  bottom,  you  can  sketch  some 
interesting  objects  or  pictures  Again  try  to  think  of  some  picture  or  object  that  no 
one  eke  will  think  of  Tty  to  make  it  teu  as  complete  and  as  interesting  a story  as  you 
can  by  adding  to  and  building  up  your  first  idea  Make  up  an  interesting  title  for  each 
of  your  drawings  and  write  it  at  the  bottom  of  each  block  next  to  the  number  of  the 


[ fwm>  enwlty  2 

QHSHaBiaaa 


Figure  3.  Picture  Completion:  Examinee  completes 
("sketch  some  interesting  objects  or  pictures")  and 
labels  a series  of  drawings  using  specific  starting 
elements.  Examinees  can  move  among  the  various 
drawing  tasks,  all  choices,  sequences,  tool  choices 


Show  instructions! 


Rctiuity  3.  CIRCLES 


f Tooto| 


✓ 


& 


a.i» 


Move 


ooooo 

ooooo 

ooooo 


O 2 O 

90  frtvlous  9*  «•*» 

Figure  4.  Circles  how  many  objects  or  pictures  can 
be  made  from  the  circles.  Can  draw  inside  and/or 
outside  the  circles.  Followed  by  a second  large 
screen  of  additional  circles. 


DRTR2:  Rctiuity 2 


i A2nme(r«w) 

fl,  56325 
f,  56327 
fl,  265780 
f,  265784 
pi,  268507 
fl,  287096 
f2,  287695 
f6,  288  351 
fH,  79061/ 
f5,  291001 
no,  294567 
f,  294568 
HO,  36462  3 
f,  364627 
no,  378131 
f,  378134 
no,  3828S4 
f,  382854 
pi,  479300 
el, 480441 
pi,  48781S 


n,  2 

r,  209453 
fl,  4 
f,  2723 
pi, 18589 
n,  599 
17.  656 
f6,  2266 
fH.  384 
fS,  3566 
no,  1 


- Response  Latencies 

- Time  using  each  tool 

|-  Nurrvoer  of  tool  switches 
Title  corrections 
Time  spent  in  each  draw 

- Seqences 


Figure  6.  All  the  data  for  the  series  of  drawings  in 
activity  two  are  collected  and  summarized  on  one 
card. 


and  labeling  and  their  timings  are  recorded 


899 


DRTRI:  Hctiuity  I 


Student  ID: 

Date:  4/3/96 

Time:  1:49  PM 

Name:  


Wtiat.lu.mflasMrg 

- Response  Latency 

- Time  using  each  tool 

- Mumper  of  tool  switches 

- Title  corrections 


Demographic 

Altime(raw) 

A1  title 

V 

5 

*,  3082/0 
p,  308692 

e,  TOVHOI 
t.  110162 

p,  311010 
n,  311442 

f,  31S77S 
f,  345571 

- 

2 

missing 

sumethmgmissi 
ng  something 

s> 

2 

(calculate  each  tool  lint 

calculate  sum  ) 

»,  422 
P.  1109 
e,  361 
c,  848 
p.  432 
m.  4333 

5 

2 

Latency,  422 
Pencilsum,  1541 
Hrasrsum,  361 
Titlesum,  848 
Movesum,  4333 
Toolswitch,  S 

5 

2 

Figure  5.  Data  collection  card  for  activity  one:  all 
timing  and  sequences  are  recorded  in  fields  and 


DRTR3:  Hctiuity 3 


What  to  measure 

- Response  Latency 

- Time  using  each  tool 

- Number  of  tool  switches 

- Title  corrections 


Figure  7.  Similarly,  the  data  for  activity  three  are  stored 


summarized. 


and  summarized 


Expanded  Drawing  Resource 


The  following  format  was  developed  to  provide  a means  of  monitoring  the  progression  of  the  examinees'  drawing  at 
regular  intervals.  This  illustration  is  a general  prototype  that  can  be  applied  to  any  test  item  format  to  create 
permanent  recordings  of  the  stages  of  completion. 


«. ►.«>!  * Ill  • 


Activity  1.  Picture  Construction 

When  you  begin  you  wall  err  a thape  in  4 box  You  arr  to  think  of  a pinurr 
or  an  object  which  you  can  draw  using  this  shape. 

Try  to  think  of  a picture  that  no  one  else  wdl  think  of  Keep  adding  nw 
id»ac  ro  your  flrci  idea  to  make  >t  ted  at  tnerecr.ng  and  at  exciting  a trory 
at  you  can.  When  you  have  completed  your  pinur,  think  up  a name  or  title 
for  k and  write  it  at  the  bottom  of  the  page  in  the  space  provided.  Make 
your  title  as  dever  and  unusual  as  possible.  Use  A to  help  tell  your  story 


Figure  8.  The  task  begins  with  instructions  on  what 
the  task  is  and  how  to  complete  it.  When  the 
examinee  is  ready  to  begin  a click  on  the  begin 
button  removes  the  text  and  reveals  the  drawing 
interface.  This  also  starts  a timer  to  record  the 
current  times  and  to  begin  the  screen  capture 
sequence. 


Figure  9.  The  drawing  instructions  are  repeated.  The 
examinee  is  presented  a button  for  drawing  and 
erasing.  There  is  a box  for  entering  a title  for  the 
drawing.  All  actions  are  recorded  and  timed  and 
snapshots  of  the  drawing  area  are  taken  at  ten  second 
intervals  until  the  ’Done'  button  is  clicked. 
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Figure  10.  Viewing  the  screen  captures.  The 
sequence  of  screen  captures  is  loaded  into  the  viewer 
by  simply  dragging  the  folder  from  the  desktop  onto 
the  file  window  of  the  viewing  utility. 


o f>  n 


Imriif*  Vtubhui 


SeuctefoWf*  t -trgavutf 


-e- 


Figure  11.  The  sequence  of  captured  screens  are 
then  accessed  using  the  scrubbing  control.  The 
current  screen  is  shown  in  the  center  with  its  time 
stamp.  The  previous  and  next  graphic  in  the 
sequence  are  displayed  to  the  right  and  left.  A text 
area  is  provided  for  entering  notes  or  annotations  for 
each  drawing  sequence.  The  text  can  be  consolidated 
for  a final  report. 
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The  Effects  of  Various  Animation  Strategies  in  Facilitating  the 
Achievement  of  Students  on  Tests  Measuring  Different  Educational 

Objectives 
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Abstract 

The  purpose  of  this  quantitative  study  was  to  investigate  the  instructional  effects  of  various 
animation  strategies.  Three  treatment  groups  involved  include  static  graphic,  animation  used  as  an 
attention -gaining  strategy,  and  animation  used  as  an  attention -gaining  and  elaboration  strategy. 
Participants  were  115  college  student  volunteers.  Two-way  multiple  analysis  of  variance  (MANOVA)  was 
used  to  analyze  the  data.  Important  findings  indicate  equivalent,  non  significant  difference  in  performance 
between  high  and  low  prior  knowledge  participants.  This  was  so  regardless  of  animation  strategies.  The 
effects  of  the  animation  strategies  on  the  achievement  of  students  with  low  prior  knowledge  are  discussed 
in  detail. 


Introduction 

The  advancement  of  technologies  make  it  possible  to  present  computer-based  multimedia 
instruction  that  includes  motion,  voice,  data,  text,  graphics,  and  still  images  (Moore,  Burton  & Myers, 

1996).  One  important  combination  of  media  is  animation,  that  is,  images  in  motion  (Dwyer  & Dwyer, 
2003).  Animation  has  been  the  focus  of  recent  attention  and  interest  and  become  more  and  more  popular. 
Although  animation  seems  to  attract  learners’  attention  and  increase  their  motivation  to  learn,  whether  or 
not  instructions  using  animation  strategies  can  facilitate  learning  still  remains  a question.  This  study 
attempted  to  examine  the  effects  of  two  specific  animation  strategies  on  student  achievement. 

Related  Literature 

Information  Processing  & Dual  Coding 

Many  early  information  processing  theories  described  a human  brain  as  being  similar  to  a 
computer,  and  human  learning  as  being  similar  to  how  computer  processes  information.  There  are  three 
main  storage  structures  in  the  memory  system:  sensory  register,  which  registers  stimuli  in  the  memory 
system;  short-term  memory  (STM),  which  serves  as  temporary  storage;  and  long-term  memory  (LTM) 
where  information  is  permanently  stored.  Short-term  memory  can  only  hold  five  to  nine  chunks  of 
information  (Miller,  1956)  before  it  is  processed  in  LTM.  Not  all  the  information  stored  in  the  LTM  can  be 
retrieved.  Retrieval  is  more  likely  when  appropriate  cues  are  provided  in  the  encoding  process  (Driscoll, 
1994). 

Pavio’s  (1986)  dual  coding  theory  further  stated  that  there  are  two  separate  information  processing 
systems:  a visual  system  which  processes  visual  knowledge  and  a verbal  system  for  processing  verbal 
knowledge.  Animation,  because  of  its  unique  dynamic  function,  is  more  likely  to  be  coded  as  both  visual 
and  verbal  knowledge  and  stored  into  long-term  memory.  Therefore,  animation  strategies  should  facilitate 
encoding  and  retrieval  process  (Paivio,  1986;  Rieber,  1994). 

Dual  coding  theory  also  suggests  there  are  three  distinctive  levels  of  processing  that  can  occur 
between  the  verbal  and  visual  system:  representational,  associative  and  referential  (Rieber  1996). 
Representational  processing  connects  the  incoming  stimuli  from  the  environment  to  either  the  verbal  or 
visual  system.  Associative  processing  constructs  connections  within  either  of  the  verbal  or  visual  systems, 
and  referential  processing  builds  connections  between  the  verbal  and  visual  systems  (Rieber).  In  this  study, 
static  graphics  facilitate  representational  processing  by  providing  the  illustrations.  Animation  as  an 
attention-gaining  strategy  facilitates  associative  processing  by  highlighting  specific  parts  of  the  heart  using 
animated  arrows.  Animation  as  an  attention-gaining  and  elaboration  strategy  facilitates  both  the  associative 
and  referential  processing  by  building  the  connections  between  the  animation  graphics  and  the 
correspondent  texts. 
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Animation  as  an  Aid  to  Information  Processing 

Animation,  with  its  unique  dynamic  function,  is  expected  to  facilitate  the  learner  encoding  the 
information  into  long-term  memory  by  providing  a “deeper”  and  “harder”  encoding  process  than  static 
visuals  (Lin,  2001).  Reiber,  Boyce,  and  Assad  (1990)  suggested  that  “although  animation  did  not  affect 
learning,  it  helped  decrease  the  time  necessary  to  retrieve  information  from  long-term  memory  and  then 
subsequently  reconstruct  it  in  short-term  memory”  (p.  50).  Reiber  (1990)  further  explained  that  animations 
facilitate  the  reconstructing  process  during  retrieval  by  encouraging  organization. 

One  animation  strategy  used  in  this  study  was  attention-gaining.  Rieber  (1990)  pointed  out 
attention-gaining  is  one  of  the  three  major  functions  of  animation.  Attention-gaining  animations  provide 
additional  ways  to  insure  selective  perception  where  specific  features  of  the  presentation  are  emphasized, 
stored  and  processed  in  the  STM  (Gagne,  1985).  Similarly,  Hannafin  and  Peck  (1988)  suggested  that 
animations  can  help  emphasize  important  information  by  providing  contrast  to  the  static  background.  In 
addition.  Levin,  Anglin,  and  Carney  (1987)  argued  attention-gaining  graphics  can  help  make  relationships 
between  ideas  more  apparent  by  facilitating  organization. 

Another  strategy  used  in  this  study  is  animation  for  elaboration.  According  to  E.  Gagne  (1985), 
“elaboration  is  the  process  of  adding  to  the  information  being  learned”  (p.  83).  Elaboration  can  have  many 
forms:  a logical  inference,  a continuation,  an  example,  a detail,  or  anything  else  that  serves  to  connect 
information.  She  further  stated  that  elaboration  facilitates  retrieval  because  it  provides  alternative  pathways 
and  extra  information  to  generate  answers. 

This  study  investigates  the  effects  of  animation  as  attention-gaining  and  elaboration  strategies  in 
facilitating  students’  achievement.  Animated  arrows  which  direct  learners’  attention  to  specific  image  parts 
were  used  as  an  attention-gaining  strategy  to  help  arouse  student  interest  as  well  as  help  them  attend  to 
relevant  cues  or  details  provided  by  animation.  Animated  text  prompts  were  used  as  an  elaboration  strategy 
to  add  extra  new  information  to  students’  existing  knowledge. 

A Model  of  Animation,  Dual -coding,  and  Information  Processing 

Gagne  and  Driscoll  (1988)  created  a basic  model  of  learning  and  memory  underlying  modern 
information  processing  theories.  It  was  revised  to  show  how  animation  works  as  an  aid  to  dual-coding  and 
information  processing  (see  Figure  1).  Humans  process  visual  and  verbal  information  from  the  environment 
simultaneously.  Animation  is  processed  as  a part  of  the  visual  information.  Animation  as  an  attention- 
gaining  strategy  helps  to  gain  attention  and  reduce  the  processing  demands  in  STM,  while  animation  as  an 
elaboration  strategy  not  only  helps  reduce  the  processing  demands  in  STM,  but  also  facilitates  encoding 
and  retrieval  processes  by  connecting  information  and  providing  alternative  retrieval  pathways  (E.  Gagne, 
1985). 
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Figure  1 . A model  of  Animation,  Dual-Coding  and  Information  Processing 
Revised  from  “The  Basic  Model  of  Learning  and  Memory  Underlying  Modern  Information 
Processing  Theories,”  by  R.  M.  Gagne  and  M.  P.  Driscoll,  1988,  Essentials  of  learning  for 
instruction,  p.  13. 

Animation  Research 

The  previous  research  on  animations  in  CBI  has  showed  somewhat  mixed  results.  Rieber  (1990) 
reviewed  13  empirical  studies  investigating  the  effect  of  animation  in  CBI  and  found  only  5 of  them 
showed  a significant  effect  for  the  animated  treatments,  while  8 showed  insignificant  differences.  Park  and 
Hopkins  (1993)  summarized  25  studies  investigating  the  effects  of  dynamic  versus  static  visual  displays. 
Fourteen  of  the  studies  found  significant  effects  for  dynamic  visual  displays.  Reiber  (1990)  suggested  the 
reason  for  the  mixed  results  could  be  rooted  in  the  procedural  flaws  of  the  previous  research,  or  maybe  it 
was  because  animations  were  not  used  in  locations  where  they  were  necessary. 

More  recent  animation  research  has  been  conducted  which  use  the  same  content  as  this  study. 
Wilson  (1998)  tested  four  types  of  treatment  groups:  still  graphics,  progressive  reveal,  animation,  and 
animation  and  progressive  reveal.  Haag  (1995)  conducted  a study  which  included  the  following  treatment 
groups:  control  group,  visual  summary  with  manipulation,  learner- manipulation  and  computer 
manipulation  group.  Lin  (2001)  proposed  using  additional  instmctional  strategies,  and  the  treatment  groups 
in  his  study  were : static  visual,  animated  only,  animation  with  advance  organizers  and  animation  with 
adjunct  questions  and  feedback.  Owens  (2002)  used  three  treatment  groups:  animation,  animation  and 
attention-directing  strategies,  and  animation  and  visual-elaborating  strategies.  The  results  of  these  studies 
showed  insignificant  differences  in  students’  achievement  among  the  treatment  groups. 

In  sum,  previous  animation  research  shows  mixed  results  while  current  animation  research 
suggests  insignificant  differences  for  treatments  incorporating  animation  strategies. 

Prior  Knowledge 

Prior  knowledge  has  been  considered  the  most  important  single  factor  that  influences  learning 
(Ausubel,  1968).  lonassen  and  Grabowski  (1993)  defined  prior  knowledge  and  achievement  as  the 
knowledge,  skills  or  abilities  that  the  learners  brings  to  the  learning  environment  before  the  instruction. 
Dwyer  (1994)  further  classified  students’  prior  knowledge  into  high,  medium  and  low  level.  Hannafin 
(1997)  suggested  that  compared  to  individuals  who  have  lower  prior  knowledge,  individuals  who  have 
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higher  prior  knowledge  can  quickly  determine  their  own  learning  needs,  generate  their  own  learning 
strategies,  and  assimilate  new  information  to  their  existing  knowledge  structure . Rieber  (2000)  also  stated 
related  prior  knowledge  provides  the  learners  unique  relevant  elaboration  that  is  unavailable  to  learners 
with  limited  prior  knowledge.  It  is  suggested  that  knowledge  will  be  encoded  more  meaningfully  and 
retrieved  more  easily  by  learners  with  high  prior  knowledge. 

Mayer  and  Anderson  (1992)  found  that  learning  significantly  improved  for  students  who  possess 
low  prior  knowledge  when  verbal  and  visual  information  are  presented  simultaneously.  They  suggested  that 
experienced  students  might  be  able  to  build  referential  connections  between  verbal  and  visual  information 
and  their  existing  knowledge  on  their  own.  The  computer-based  instruction  utilized  in  this  study  presented 
verbal  (the  text)  and  visual  (the  graphic  illustration  or  animation)  information  simultaneously.  One  of  the 
purposes  of  this  study  is  to  investigate  if  varied  animation  strategies  will  improve  the  performance  of  the 
students  identified  as  possessing  low  levels  of  prior  knowledge. 

Research  Purpose  and  Questions 

In  this  regard,  the  purpose  of  this  research  was  to  investigate  the  instructional  effect  of  various 
animation  strategies  on  facilitating  achievement  of  college  level  students  with  high  and  low  levels  of  prior 
knowledge.  Three  research  questions  were  explored. 

Do  various  animations  used  to  gain  attention  or  to  gain  attention  and  elaborate  on  the  content 
improve  students’  performance  on  tests  measuring  different  types  of  educational  objectives? 

Do  various  animations  used  to  gain  attention  or  to  gain  attention  and  elaborate  on  the  content 
improve  the  performance  of  the  students  identified  as  possessing  high  and  low  levels  of  prior  knowledge  on 
tests  measuring  different  educational  objectives? 

Is  there  an  interaction  between  levels  of  prior  knowledge  and  the  selected  animation  strategies? 

Methodology 

Participants 

One  hundred  and  fifteen  student  volunteers  participated  in  the  study.  Most  were  freshmen.  Fifty 
eight  were  classified  as  high  prior  knowledge  participants  while  57  were  classified  as  low  prior  knowledge 
participants.  Four  participants  did  not  complete  the  study. 

Instructional  Materials 

The  self-paced  web-based  instruction  used  in  this  study  was  adapted  from  paper-based  text 
materials  developed  by  Dwyer  and  Lamberski  (1997)  about  the  human  heart . The  original  script  of  the 
heart  content  contains  approximately  1,800  words  divided  into  three  sections:  the  parts  of  the  heart, 
circulation  of  blood  and  cycle  of  blood  pressure.  Integration  and  positioning  of  the  animation  strategies  was 
determined  by  an  item  analysis  which  identified  where  students  were  having  difficulties  based  on  their 
performance  on  the  criterion  tests  from  a previous  pilot  study  conducted  in  Summer,  2003. 
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Treatments 

Static  graphic  ( Control  group,  T1 ):  This  treatment  contained  one  page  of  directions  and  twenty 
pages  of  instructional  screens  with  instructional  text  on  the  left  and  the  correspondent  static  graphic  on  the 
right.  An  example  screen  is  shown  in  Figure  2. 


SEPTUM 


APEX 


Figure  2.  Example  screen  of  static  graphic  treatment 

Animation  as  attention-gaining  strategy  (T2):  This  treatment  is  the  same  as  T1  except  that  thirteen 
instructional  screens  contained  embedded  attention-gaining  animations.  The  other  seven  screens  contained 
only  static  graphics  since  previous  item  analysis  did  not  indicate  students  have  difficulties  with  those  items. 

Thirteen  screens  contained  animation  with  the  static  graphic  on  the  right  of  the  screen  and  a “Click 
to  See  the  Animation”  button  below.  Animated  arrows  were  used  as  an  attention-gaining  strategy  to  direct 
students’  attention  to  specific  parts  of  the  heart.  In  order  to  lower  the  cognitive  load  of  the  students, 
animations  were  grouped  into  chunks.  When  one  animation  was  finished,  a “Continue”  button  appeared. 
The  students  then  clicked  on  the  “Continue”  button  to  see  the  next  animation.  After  all  the  animations  were 
shown,  the  static  graphic  was  restored  and  a set  of  three  buttons  appeared:  "BACK”,  “NEXT”,  and  “Replay 
the  Animation.”  See  example  screen  in  Figure  3. 
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Frame  3 

Figure  3.  Example  screen  of  animation  as  attention-gaining  strategy  treatment 

Animation  as  attention-gaining  and  elaboration  strategy  (T3):  this  treatment  is  also  the  same  as  T1  except 
that  thirteen  instructional  screens  contained  embedded  attention-gaining  and  elaboration  animations.  A 
pop-up  animation  that  highlighted  the  most  important  information  in  this  instructional  screen  was  used  in 
combination  with  the  animated  arrows.  See  example  screen  in  Figure  4. 
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Frame  3 

Figure  4.  Example  screen  of  animation  as  attention-gaining  and  elaboration  strategies  treatment 

Procedures 

The  two -stage  study  included  an  online  pretest  to  classify  the  participants  into  high  and  low  prior 
knowledge  and  a following  lab  session.  The  mean  of  the  human  physiology  pretest  scores  (57.55)  of  the 
original  1 15  participants  was  used  as  a cut  point  to  distinguish  between  high  or  low  prior  knowledge 
individuals,  who  were  then  systematically  assigned  to  one  of  the  three  treatment  groups. 

There  were  111  participants  who  went  to  the  lab  session.  The  participants  were  instructed  to  go  to 
a specified  URL  for  the  treatment.  Afterwards,  the  participants  took  four  criterion  tests:  the  drawing  test, 
identification  test,  terminology  test  and  comprehension  test. 

Criterion  Measures 

The  criteria  measures  included  a 20-item  paper-based  drawing  test,  and  three  separate  20-item 
multiple -choice  web-based  criterion  tests:  identification  test,  terminology  test  and  comprehension  test. 

In  all  the  four  criterion  tests,  the  test  reliabilities  were  all  above  0.8  (Drawing  = .89;  Identification 
= .86;  Terminology  = .82;  Comprehension  = .82),  which  is  the  satisfactory  reliability  suggested  by  Anastasi 
and  Urbina  (1997). 

Data  Analysis 

SPSS  was  used  to  analyze  the  data.  The  study  used  a 2 X 3 factorial  design  with  two  levels  of 
prior  knowledge  and  three  animation  strategies.  A two-way  MANOVA  was  used  to  test  for  the  main  effects 
and  the  interaction  between  the  prior  knowledge  level  and  the  three  treatments . 


Results 


Analysis  of  the  Physiology  Pretest 

The  scores  of  the  human  physiology  pretest  were  converted  into  percentages.  They  ranged  from  33 
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to  78  with  a mean  of  57.55.  A one-way  ANOVA  was  conducted  to  determine  if  there  was  a difference 
among  the  three  treatment  groups  in  the  means  of  their  prior  knowledge.  The  p-value  from  the  ANOVA 
(p>.05)  indicated  that  the  three  groups  were  not  significantly  different  in  terms  of  their  prior  knowledge. 

Descriptive  Statistics 

Table  1 below  shows  the  means  and  standard  deviation  for  the  four  criterion  tests  and  their 
combined  total  by  treatments  and  levels  of  prior  knowledge.  They  showed  that  the  mean  scores  for  the  high 
and  low  prior  knowledge  participants  in  each  treatment  group  were  similar. 

Table  1 Means  and  Standard  Deviation  for  Four  Tests  by  Treatment  and  Levels  of  Prior  Knowledge 


Prior 

Knowledge 

Level 

N 

Drawing 

Identification 

Terminology 

Comprehension 

Total 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Static  Graphic  Group  (Tl) 

Low 

19 

14.63 

4.90 

15.16 

3.95 

11.68 

4.61 

10.84 

3.89 

52.31 

17.35 

High 

19 

13.47 

5.56 

16.16 

3.25 

13.00 

4.06 

12.05 

4.65 

54.68 

17.52 

38 

14.05 

5.23 

15.66 

3.60 

12.34 

4.34 

11.45 

4.27 

53.50 

17.44 

Animation  as  Attention-gaining  Strategy  (T2) 

Low 

16 

11.69 

6.19 

15.44 

5.10 

9.50 

5.16 

9.25 

5.45 

45.88 

21.90 

High 

19 

12.95 

4.37 

13.89 

5.05 

11.63 

4.65 

10.68 

4.35 

49.15 

18.42 

35 

12.32 

5.28 

14.67 

5.08 

10.57 

4.91 

9.97 

4.90 

47.52 

20.16 

Animation  as  Attention-gaining  and  Elaboration  Strategy  (T3) 

Low 

19 

14.26 

4.95 

13.84 

4.61 

12.53 

4.70 

11.89 

4.33 

52.52 

18.59 

High 

19 

14.74 

4.86 

14.32 

4.22 

11.74 

4.40 

11.79 

4.72 

52.59 

18.20 

38 

14.50 

4.91 

14.08 

4.42 

12.14 

4.55 

11.84 

4.53 

52.56 

18.40 

Note:  Drawing,  Identification,  Terminology  and  Comprehension  scores  could  range  from  a low  of  0 to  a 
high  of  20. 


The  results  of  an  ANOVA  for  the  sum  means  showed  that  the  differences  in  variance  were  not 
significant  among  Tl,  T2  and  T3,  F(2,108)=.957,  p=.387. 

Table  2 below  shows  the  means  and  standard  deviation  for  the  four  criterion  tests  scores  for  the  23 
questions  identified  in  the  pilot  study  that  the  students  have  difficulties  with.  Maximum  possible  score  is 
equal  to  the  number  of  items  on  each  test. 

Table  2 Means  and  Standard  Deviation  for  the  Four  Criterion  Tests  Scores  (23  items)  by  Treatment 


Number 

Four  Criterion  Tests  of  Items 

Static  Graphic 

Animation  as 
attention-gaining 

Animation  as 
attention-gaining  and 
elaboration 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Drawing 

4 

2.66 

1.38 

2.31 

1.37 

2.68 

1.45 

Identification 

3 

1.92 

1.15 

1.83 

1.04 

1.66 

1.12 

Terminology 

9 

4.89 

2.48 

3.83 

2.54 

4.45 

2.24 

Comprehension 

7 

2.76 

2.03 

2.89 

1.80 

3.24 

1.70 

Sum 

23 

12.23 

10.86 

12.03 

Analysis  of  Null  Hypothesis 

A two-way  MANOVA  was  run  to  test  the  research  question.  Two  prerequisites,  equality  of 
variances  and  the  correlations  between  the  dependent  variables,  were  checked  before  the  MANOVA  was 
used.  In  all  cases,  except  the  identification  tests,  the  Pearson  correlation  coefficient  is  .6  or  higher  at  the 
0.01  level.  A two-way  ANOVA  was  conducted  for  the  identification  test  to  check  if  there  were  any 


910 


differences. 

Table  3 below  showed  the  overall  MANOVA  Results  using  Pallai’s  Trace  F. 

Table  3 Summary  of  Two-Way  MANOVA  Results  of  Four  Achievement  Measures  (80-items)  by  Treatment 
Group  and  Prior  Knowledge  Levels  ( Low,  or  High ) 


Source 

Pillai’s  Value 

F 

dfl 

df2 

P 

Treatment 

.074 

.990 

8 

206 

.445 

Prior  Knowledge 

.017 

.452 

4 

102 

.771 

Interaction 

.079 

1.053 

8 

206 

.398 

The  results  of  Pillai’s  Trace  in  a two-way  MANOVA  analysis  showed  that: 

There  were  no  significant  differences  among  the  three  animation  strategies  on  any  of  the  criterion 
tests,  F(8,206)=.990,  p=.445.  Therefore,  the  null  hypothesis  1 was  retained 

There  were  no  significant  differences  between  students  with  high  and  low  prior  knowledge  on  any 
of  the  criterion  tests,  F(4,102)=.452,  p=.771.  Therefore,  as  predicted,  the  null  hypothesis  2 was  retained. 

There  was  no  significant  interaction  between  the  two  levels  of  prior  knowledge  and  three  types  of 
treatments,  F(8,206)=1.053,  p=.398.  Therefore,  the  null  hypothesis  3 was  retained. 

Further  analysis  (See  Table  4)  was  conducted  for  the  23  questions  identified  in  the  pilot  study  that 
the  students  were  having  difficulties  with. 

Table  4 Summary  of  Two-  Way  MANOVA  Results  of  Four  Achievement  Measures  (23-items)  by  Treatment 
Group  and  Prior  Knowledge  Levels  ( Low,  or  High ) 


Source 

Pillai’s  Value 

F 

Dfl 

Df2 

P 

Treatment 

.071 

.945 

8 

206 

.480 

Prior  Knowledge 

.007 

.184 

4 

102 

.946 

Interaction 

.045 

.591 

8 

206 

.785 

Again,  based  on  the  above  results,  all  the  hypotheses  were  retained. 

Conclusions  and  Discussion 

Given  that  prior  knowledge  has  been  considered  the  most  important  single  factor  that  influences 
learning  (Ausubel  1968),  the  findings  that  low  prior  knowledge  students  perform  equally  well  as  those  high 
prior  knowledge  students  become  important.  Given  that  creating  animation  is  also  time  -consuming  and 
costly,  these  findings  add  to  the  growing  literature  supporting  the  power  of  static  graphics. 

Static  Graphic  vs.  Animation  Strategies 

This  study  attempted  to  examine  different  animation  strategies  from  the  previous  research. 
Insignificant  differences  were  found  between  the  groups  using  animation  strategies  and  the  control  group 
using  static  graphics.  The  results  showed  that  the  static  graphics  group  performed  equally  as  well  as  the 
animation  strategies  group.  This  overall  finding  continues  the  debate  about  the  value  of  animation  versus 
just  providing  visualization.  Visualization,  included  in  all  treatments  seemed  to  be  a powerful  factor  in 
learning  this  material.  The  results  were  in  accordance  with  many  previous  literature  and  animation-related 
studies.  Mayer  (1997)  justified  the  effect  of  using  coordinated  presentation  of  explanation  in  visual  format 
(illustrations).  Wilson  (1998)  found  a general  tendency  of  the  mean  score  for  the  static  treatment  produce 
somewhat  better  results  than  any  of  the  dynamic  treatments.  Owens  (2002)  found  a trend  that  the  students’ 
performance  decreased  as  animation  strategies  were  added  to  the  instructional  screens. 

Theoretically,  the  results  of  the  study  strengthened  the  results  and  conclusions  of  some  of  the 
previous  animation-related  studies.  Practically,  the  results  also  raised  a very  important  question  to  the 
practice  of  instructional  designer,  it  is  it  really  worth  it  to  design  and  develop  instructions  utilizing 
animation  strategies  versus  simply  using  static  graphics  if  static  graphics  have  been  shown  to  be  at  least  as 
effective  as  animation?  As  we  all  know,  static  graphics  are  more  cost-effective  and  cost-efficient  than 
animations.  In  future  design,  maybe  it  is  better  to  utilize  static  graphics  as  much  as  possible  and  use 
animations  only  when  the  use  of  animation  is  justified  (Rieber,  1990). 
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High  vs.  Low  Prior  Knowledge 

The  result  of  the  interaction  between  level  of  prior  knowledge  and  strategy  use  also  provides  an 
important  contribution  to  the  debate  about  the  effectiveness  of  animation.  What  this  study  showed  was  that 
students  with  lower  prior  knowledge  performed  equally  well  to  those  with  high  prior  knowledge  in  all  three 
treatments.  This  result  was  contrary  to  much  previous  research  that  showed  high  prior  knowledge  students 
performing  better  than  low  prior  knowledge  students  regardless  of  treatment.  We  believe  this  can  be 
explained  by  dual  coding  theory.  Students  with  low  prior  knowledge  are  helped  more  when  verbal  and 
visual  informatbn  are  presented  simultaneously  since  it  helps  them  build  referential  connections  (Mayer  & 
Anderson,  1992).  In  this  study,  by  rearranging  the  layout  of  the  instructional  text  and  static  visuals,  the 
static  graphics  or  the  animations  were  put  side  by  side  instead  of  static  graphics  on  the  top  and  instructional 
text  at  the  bottom.  Based  on  the  previous  literature,  the  researchers  believed  that  this  layout  would 
encourage  the  learners  to  read  the  instructional  text  as  well  as  build  connections  with  the  static  graphic  or 
animations.  There  was  a significant  difference  between  the  high  and  low  prior  knowledge  participants  in 
the  pretest,  but  the  differences  were  obviously  reduced  to  insignificant  differences  in  the  four  achievement 
tests  after  they  went  through  the  treatments. 

Prior  Knowledge  and  Treatments 

The  results  also  showed  that  there  was  an  insignificant  interaction  between  levels  of  prior 
knowledge  and  the  instructional  treatments,  different  from  the  predictions  of  the  researchers.  It  was 
expected  that  the  animation  as  an  attention-gaining  and  elaboration  strategy  group  would  perform  better 
than  animation  only  as  an  attention-gaining  strategy,  which  would  be  better  than  the  static  graphic  group 
(control  group).  It  was  also  predicted  that  the  high  prior  knowledge  participants  would  perform  better  than 
lower  prior  knowledge  participants  in  each  treatment. 

One  possible  explanation  is  that  the  animations  as  an  attention-gaining  strategy  were  attracting  the 
students  ’ attention  to  the  animation  itself  instead  of  to  the  instructional  content.  The  repeated  single 
movement  of  the  animated  arrows  may  have  bored  the  students.  Or  the  students  just  simply  were  not 
motivated  enough  to  participate  the  study  since  their  performance  would  not  affect  their  course  grade. 

In  addition  to  the  above  reasons,  another  important  reason  may  be  that  for  college  students,  static 
graphics  are  effective  enough  to  facilitate  referential  connection  between  the  verbal  and  visual  information, 
making  animation  unnecessary. 

To  sum,  the  findings  indicated  that  static  graphics  were  as  effective  as  animation.  These  results 
imply  that  it  will  be  more  beneficial  to  use  correspondent  static  graphics  more  in  our  instructions. 

Limitations 

The  results  of  the  study  are  limited  to  the  population  of  undergraduate  students  with  similar 
characteristics  (e.g.  prior  knowledge,  field-dependence/field-independence,  etc).  Further,  the  study  used 
systematic  randomization  to  assign  students  to  different  treatment  groups  according  to  their  pretest  scores 
instead  of  strictly  stratified  randomization.  Therefore,  these  generalizations  should  be  interpreted 
cautiously. 


Further  Research  Suggestions 

Future  studies  may  rerun  this  research  with  more  generalized  population  and  use  strictly  stratified 
randomization.  Secondly,  the  effects  of  other  animation  strategies,  such  as  animations  as  practice  and 
feedback  strategies,  need  to  be  explored.  Lastly,  future  studies  may  assess  the  effectiveness  of  animation 
strategies  on  facilitating  high  order  thinking,  such  as  problem-solving. 
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The  Florida  Online  Reading  Professional  Development  (FOR-PD)  program  is  funded  by  the 
Florida  Department  of  Education  (DOE)  and  is  housed  at  the  University  of  Central  Florida  (UCF).  FOR- 
PD  is  an  online  staff  development  project  designed  to  help  teachers  improve  reading  instruction  for  learners 
in  grades  preK-12.  Developed  collaboratively  with  literacy  and  technology  experts,  school  districts, 
professional  organizations,  and  teacher  educators  across  the  state  of  Florida,  the  project  functions  as  a 
primary  statewide  delivery  mechanism  for  improving  teaching  methods  in  reading  instruction  to  preK-12 
teachers. 

As  indicated  in  the  Request  for  Proposal  (RFP)  from  the  Florida  Department  of  Education  (FL 
DOE),  for  example,  “feedback  and  leader-peer  response”  and  “monitoring  of  assignments”  were  considered 
essential.  In  response  to  the  RFP,  the  UCF  proposal  for  FOR-PD  highlighted  the  role  of  facilitators, 
asserting  in  the  grant  proposal  text,  “Feedback  by  facilitators  is  critical  to  the  performance  of  participants.” 
In  addition,  the  discussion  boards  were  also  depicted  as  critical  factors  for  successful  delivery  of  online 
learning  by  both  documents. 

Prior  research  has  emphasized  facilitator’s  roles  in  online  education  or  training  courses, 
“facilitating  online  dialogue,  community,  and  ultimately,  education”  (Collison,  Elbaum,  Haavind,  & 

Tinker,  2000).  Meanwhile,  Lieblein  (2000)  explored  the  role  of  threaded  discussion  board  as  s critical 
factor  for  successful  delivery  of  online  programs  based  on  nearly  10  years  of  academic  and  administrative 
experience  with  online  programs.  On  the  other  hand,  qualitative  analysis  could  complement  and  be  used  in 
conjunction  with  the  quantitative  methods  by  explaining  reasons  for  observed  differences  (Gunawardena, 
Lowe,  & Carabajal,  2000). 


Recruiting  Facilitators 

We  relied  on  three  core  criteria  for  selecting  facilitators  for  FOR-PD.  First,  the  facilitator  needed 
to  have  strong  content  knowledge  in  reading.  Second,  we  sought  online  facilitators  who  had  experience  as 
literacy  leaders  and  literacy  experts.  Third,  we  looked  for  facilitators  who  expressed  desire  to  learn  along 
with  us  about  helping  preK-12  teachers  develop  their  reading  knowledge  and  expertise.  Given  the  novelty 
of  this  large-scale  high  profile  state  online  project,  prior  experience  with  the  Internet  was  not  mandatory. 
However,  we  knew  that  some  facilitators  were  reasonably  comfortable  in  online  learning  since  they  had 
been  involved  in  it  before  as  students  or  facilitators. 

FOR-PD  Facilitators 

FOR-PD  facilitators  play  a vital  role  in  developing  and  maintaining  an  online  professional 
development  program  that  is  effective,  efficient,  and  supports  the  realization  of  the  FOR-PD  project 
objectives.  The  primary  purpose  of  a FOR-PD  facilitator  is  to  interact  with  FOR-PD  course  participants. 
This  translates  to  encouraging  and  replying  to  email  messages  and  discussion  postings,  providing  feedback 
on  assignments,  and  being  the  "point  person"  for  answering  their  questions.  They  must  also  be  responsive 
to  individual  district  requests  and  needs. 

A facilitator  in  the  online  environment  must  possess  a unique  set  of  skills  to  perform  effectively.  Some  of 
the  basic  criteria  for  a person  to  be  successful  as  an  online  facilitator  include  the  following.  Facilitators 
must  be  able  to  create  a supportive  environment  where  all  students  feel  comfortable  participating  and 
especially  where  students  know  that  their  facilitator  is  accessible.  Facilitators  should  give  students  timely 
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quality  feedback  on  student  contributions  to  discussions,  assignments,  and  quizzes.  Facilitators  should  keep 
students  advised  of  their  progress  respect  to  the  course  evaluation  process  on  a regular  basis.  Facilitators 
should  feel  comfortable  communicating  in  writing.  The  face-to-face  contact  traditionally  available  in  a 
classroom  setting  is  not  available  in  the  online  learning  process.  The  ability  to  verbally  communicate  is 
replaced  with  a keyboard.  Facilitators  must  be  comfortable  communicating  in  writing  because  that  is  the 
fundamental  process  of  online  learning.  The  facilitator  is  the  primary  person  participants  interact  with  who 
provides  the  human  factor.  Facilitators  should  be  experienced  and  well  trained  in  online  learning.  This 
includes:  sending  and  receiving  email;  using  discussion  boards;  using  chat  tools;  using  a web  browser- 
Netscape  or  Internet  Explorer. 

Facilitator  Training  and  Certification  Course 

To  become  a certified  FOR-PD  facilitator,  interested  educators  must  complete  an  online 
application  and  possess  the  following  qualifications:  1)  successful  completion  (80%  mastery  or  above)  of 
the  FOR-PD  course;  2)  a minimum  of  three  years  teaching  experience;  3)  master’s  degree  in  reading  or 
other  related  areas;  4)  advanced  knowledge  of  research-based  reading  strategies;  5)  ability  to  provide 
explicit  instruction  in  the  following  elements  of  reading  as  they  apply  to  appropriate  grades:  phonemic 
awareness,  phonics,  fluency,  vocabulary,  and  comprehension;  6)  ability  to  systematically  use  effective 
reading  strategies  that  have  been  tested  and  have  a record  of  success;  and  7)  identified  by  school  or  district 
as  a reading/literacy  leader.  In  addition  to  these  requirements,  successful  completion  of  the  FOR-PD 
Facilitator  Training  and  Certification  Course  is  also  required.  The  FOR-PD  Facilitator  Training  and 
Certification  Course  is  a 25-hour  online  professional  development  course  intended  to  ensure  that  our 
facilitators  have  the  knowledge  and  skills  they  need  to  become  successful  online  class  facilitators  for  the 
Florida  Online  Reading  Professional  Development  Course.  The  course  consists  of  the  five  lessons  that 
encompass  an  introduction  to  FOR-PD,  details  about  the  project  and  goals  of  FOR-PD,  information  on 
online  learning,  and  support  options  to  facilitators 

Following  completion  of  the  FOR-PD  Facilitator  Training  and  Certification  Course,  facilitators 
are  expected  to  demonstrate  mastery  of  the  following  skills:  1)  describe  the  FOR-PD  course  and  the  goals 
of  the  course;  2)  identify  advantages  of  online  learning;  3)  identify  potential  disadvantages  of  online 
learning  and  describe  at  least  one  way  each  disadvantage  can  be  addressed;  4)  identify  the  role  of  the  online 
class  facilitator;  5)  describe  techniques  for  facilitating  an  online  course;  and  6)  identify  and  use  online  tools 
such  as  chat,  discussion  boards,  email,  and  grade  books. 

There  is  no  charge  to  take  the  FOR-PD  Facilitator  Training  and  Certification  Course.  An 
electronic  certificate  (pdf)  is  emailed  to  participants  upon  successful  completion,  their  district  staff 
development  office  is  notified,  and  they  are  then  added  to  the  pool  of  certified  FOR-PD  facilitators. 
Completion  of  the  FOR-PD  Facilitator  Training  and  Certification  course  and  certification  as  a FOR-PD 
facilitator  does  not  guarantee  employment  as  a facilitator.  For  the  most  part,  facilitators  are  selected  to 
facilitate  by  school  districts  from  the  pool  of  qualified  facilitators.  Many  school  districts  have  a “favorite” 
facilitator  or  two  that  they  assign  to  facilitate  again  and  again.  These  are  generally  reading  specialists, 
reading  coaches,  or  literacy  leaders  in  the  district  with  particular  knowledge  of  the  unique  qualities  of  the 
district,  its  reading  programs,  teachers,  and  student  population. 

Facilitator  Support 

Facilitators  receive  assistance  from  the  FOR-PD  office  to  support  them  in  their  roles  as  online  mentors 
for  participants.  Support  in  place  includes  a facilitator  manual,  performance  support  tools,  electronic 
newsletters,  facilitator  forums,  monthly  chats,  and  supervision. 

Facilitator  Manual 

One  of  the  most  important  documents  available  to  facilitators,  the  Facilitator  Manual,  outlines 
specific  tasks  facilitators  must  complete  before  their  section  of  the  FOR-PD  course  begins.  Such  tasks 
include  getting  access  to  the  course,  getting  participants’  names,  posting  the  first  discussion,  and  sending  a 
welcome  message  to  participants.  The  facilitator  manual  also  offers  a detailed  description  of  facilitator  tasks 
to  be  completed  throughout  the  course  including  reminders  to  monitor  participants’  progress,  grading 
assignments,  offering  feedback,  handling  course  “no  shows,”  and  those  who  fall  behind.  The  Facilitator 
Manual  offers  detailed  explanations  for  after-course  tasks  such  as  sending  a wrap-up  message,  thanking 
participants  for  their  participation,  and  instructions  for  notifying  school  district  and  FOR-PD  offices  that  the 
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course  has  ended.  The  manual  also  describes  the  various  support  mechanisms  available  to  facilitators  and 
outlines  the  responsibilities  of  key  stakeholders  including  participants,  school  districts,  and  the  FOR-PD 
office.  Conditions  of  employment  are  carefully  explained  and  tutorials  for  WebCT  chat,  discussion  boards, 
email  tools,  and  grade  book  tools  are  provided.  The  manual’s  appendix  offers  sample  messages  for  a variety 
of  purposes  including  welcoming  participants  to  the  course,  helping  those  with  login  trouble,  notice  to  those 
lagging  behind,  and  a sample  message  to  thank  participants  for  their  participation. 

Performance  Support  Tools 

FOR-PD  course  facilitators  have  access  to  a variety  of  performance  support  tools  to  help  them  in 
their  role  as  course  facilitator.  Specifically  requested  by  course  facilitators,  printable  rubrics  streamline 
assignment  grading  for  busy  facilitators  who  prefer  to  use  a paper-based  method  of  grading  course 
assignments.  The  Excel  grade  book  template  offers  a convenient  method  of  grading  for  facilitators  more 
comfortable  with  storing  grades  electronically.  Developed  at  the  request  of  course  facilitators,  model 
postings  provide  examples  for  facilitators  to  share  with  participants  as  needed  and  in  a way  controlled  by 
the  facilitators.  The  model  postings  also  help  facilitators  calibrate  their  own  grading  standards.  Some 
facilitators  have  requested  a course  completion  certificate  for  participants  in  their  sections.  This  optional 
component  is  available  to  facilitators  as  needed  through  the  Facilitator  Forum.  The  end-of-course  checklist 
serves  as  a convenient  reminder  to  facilitators  about  the  specific  tasks  that  must  be  carried  out  to  assure 
proper  in-service  credit  for  participants  and  prompt  payment  to  course  facilitators. 

Electronic  Newsletter  for  Facilitators 

Entitled  Facilitation  With  Felicity  (FFF*),  the  facilitator  newsletter  helps  keep  course  facilitators 
informed  of  the  latest  literacy  news  and  events  from  around  the  state  and  from  the  FOR-PD  project.  The 
FFF*  highlights  one  literacy  strategy  each  month  and  offers  a variety  of  tips  and  tricks  to  keep  facilitators’ 
skills  polished.  One  section  focuses  on  “fine  facilitation’’  to  highlight  and  promote  desirable  facilitator 
actions.  The  “Dear  Felicity’’  column  answers  facilitators’  questions  about  handling  problems,  dealing  with 
challenging  participants,  and  keeping  participants  involved  in  the  course.  The  FFF*  also  announces  awards, 
contests,  and  conferences  of  interest  to  facilitators  and  highlights  a wide  variety  of  literacy  and  professional 
resources  likely  to  be  useful  to  FOR-PD’s  facilitators. 

Facilitator  Forum 

Housed  on  the  FOR-PD  course  server,  the  Facilitator  Forum  is  a series  of  discussion  boards 
offering  24/7  access  for  facilitators  to  interact  with  each  other  to  share  information  and  ideas  about  the 
FOR-PD  course,  to  ask  for  help  from  others,  and  to  share  successes.  Links  to  FOR-PD  course  content,  the 
FFF*,  the  Facilitator  Manual,  and  optional  course  completion  certificate  offer  convenient  access  for 
facilitators.  Specific  discussion  areas  include  a place  to  meet  fellow  facilitators,  to  ask  for  and  offer  help, 
hints,  and  advice,  to  make  suggestions  for  an  upcoming  FFF*,  and  to  share  success  stories.  The  Facilitator 
Coffeehouse  discussion  board  enables  facilitators  to  interact  with  each  other  on  matters  unrelated  to  the 
FOR-PD  course,  but  likely  to  be  of  general  interest.  Finally,  there  is  a discussion  area  specifically  for 
facilitators  to  discuss  issues  related  to  each  of  the  14  FOR-PD  lessons. 

Monthly  Chats 

Monthly  facilitator  chats  serve  to  keep  facilitators  in  touch  with  each  other  and  the  FOR-PD  office. 
Chats  feature  discussions  with  subject  matter  experts  and  cover  a range  of  literacy-related  topics. 

Facilitators  often  discuss  the  mechanics  of  facilitating  a course,  recent  changes  to  the  course  or  participate 
in  an  open  forum  facilitated  by  FOR-PD  staff  to  answer  questions. 

Personnel 

All  FOR-PD  staff  members  assist  facilitators  in  answering  questions  or  solving  problems.  The 
FOR-PD  Help  Desk  is  available  for  technical  support  to  facilitators  and  participants  alike.  One  full-time 
staff  member  is  dedicated  to  assisting  facilitators  perform  their  duties.  The  Facilitator  Support  Specialist 
monitors  discussions  on  the  Facilitator  Forum,  providing  answers  as  necessary  and  identifying  issues 
raised  in  the  Facilitator  Forum  that  need  to  be  addressed  more  broadly  by  other  FOR-PD  staff  members 
and  sharing  suggestions  to  improve  the  project.  Additional  responsibilities  include  answering  email  from 
facilitators,  writing  the  FFF*  and  coordinating  monthly  chat  sessions,  as  well  as  answering  Help  Desk 
calls. 
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The  Facilitator  Support  Specialist  responds  to  emails  received  from  the  forpdfac  email  account 
for  day-to-day,  routine  implementation  questions,  and  is  also  available  to  support  facilitators  through  the 
duration  of  their  facilitation  experience,  including  sending  reminders  and  instructions  to  facilitators  as 
course  sections  begin  and  end.  After  the  course  has  finished,  Facilitator  Support  Specialist  sends  follow-up 
information  and  reminders  and  processes  payroll  paperwork. 

Facilitator  Supervision 

In  addition  to  the  required  training  and  certification,  facilitators  are  closely  monitored  throughout 
their  facilitation  of  the  FOR- PD  course.  Each  section  of  the  course  is  monitored  at  three  points:  1)  the 
beginning  of  the  course  (Lesson  1);  2)  the  midpoint  (Lesson  5-8);  and  3)  at  the  end  (Lesson  14).  The 
monitoring  helps  to  ensure  that  facilitators  are  creating  a supportive  environment  where  all  students  feel 
comfortable  participating  and  especially  where  students  know  that  their  facilitator  is  accessible;  giving 
students  timely  quality  feedback  on  student  contributions  to  discussions,  assignments,  and  quizzes;  and 
keeping  students  advised  of  their  progress  in  the  course  on  a regular  basis. 

The  FOR-PD  project  has  been  exceptionally  well  received  by  school  districts,  universities, 
administrators,  and  teachers  throughout  the  state.  We  believe  that  our  facilitators  and  the  training, 
certification,  and  support  that  we  provide  to  them  has  been  critical  to  the  success  of  the  FOR-PD  project. 
Currently,  FOR-PD  has  200  certified  facilitators  located  throughout  the  state.  Of  these  200,  111  reported 
having  taken  an  online  course  previously,  however  only  23  had  ever  taught  or  facilitated  an  online  course 
before  coming  to  FOR-PD.  390  sections  of  the  FOR-PD  course  have  been  offered  since  January  21,  2003 
and  139  facilitators  have  facilitated  a section  within  the  last  year. 

FOR-PD  Evaluation 

The  outside  interim  report  of  the  first  year  of  the  FOR-PD  project  and  course,  drawn  from  various 
sources  including  narrative  reports  from  facilitators,  surveys  of  participants  at  the  end  of  the  course,  and 
follow-up  telephone  interviews  with  administrators,  reported  the  following.  Over  87%  of  FOR-PD 
participants  indicated  they  would  make  changes  and/or  additions  to  classroom  reading  instruction  as  a 
result  of  FOR-PD.  Over  90%  (93%)  of  participants  indicated  that  the  value  of  reading  strategies  introduced 
in  FOR-PD  was  excellent  or  good.  Approximately  97%  of  participants  indicated  FOR-PD  was  excellent  or 
good  in  covering  the  state  and  national  reading  initiatives,  with  nearly  three-fourths  of  participants  (73%) 
indicating  FOR-PD  covered  the  reading  initiatives  to  an  excellent  degree.  Over  90%  of  participants 
indicated  that  FOR-PD  has  contributed  to  their  knowledge  of  effective  reading  theory,  research,  and 
instructional  practice  to  an  excellent  or  good  extent.  The  extent  FOR-PD  contributed  to  understanding 
student  needs  and  instructional  adaptations  for  struggling  readers  to  an  excellent  or  good  extent  was  89% 
with  over  one-half  indicating  excellent  (52%).  Over  90%  of  participants  rated  the  support  from  their 
facilitator  as  excellent  (74%)  or  good  (17%). 

For  the  purpose  of  the  qualitative  evaluation  of  FOR-PD  in  phase  II  of  the  project,  internal  and 
external  documents  were  reviewed.1  Hundreds  of  pages  of  qualitative  data  were  collected  and  analyzed 
from  the  discussion  boards  of  facilitators,  with  a focus  on  what  contributions  FOR-PD  facilitators  have 
made  to  the  program,  how  they  have  experienced  the  program,  and  particularly,  how  they  like  the  changes 
and  revision  of  the  FOR-PD  course  since  summer  2003. 

FOR-PD  Pilot  Summary  Report 

The  FOR-PD  Course  was  piloted  September  16-27,  2002.  The  pilot  summary  report  focused  on 
feedback  from  participants  and  concluded  that  “Overall,  the  feedback  from  the  participant  was  extremely 
positive . . .Teachers  indicated  that  the  course  provided  aspects  that  they  would  utilize  in  their  classroom 
instruction”.  This  interim  report  drew  from  various  sources  including  narrative  reports  from  facilitators, 
surveys  of  participants  at  the  end  of  the  course,  and  follow-up  telephone  interviews  with  administrators. 

FOR-PD  Courses,  Participants  and  Facilitator’s  Discussion  Board 

The  first  sections  of  the  FOR-PD  course  were  launched  on  January  21,  2003.  A total  of  4,570 
participants  had  enrolled  and  2,045  had  completed  FOR-PD  prior  to  beginning  phase  2 (September  2003). 
Of  this  number,  4,504  participated  as  employees  of  Florida  school  districts  and  66  participated  through  a 
Florida  college  or  university.  The  latter  participants  may  include  students  and  college/university  faculty. 
During  fall  2003,  approximately  2,200  additional  students  enrolled  in  FOR-PD. 

In  February  2003,  a Facilitator  Forum  was  established  using  the  WebCT  discussion  board.  The 
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primary  purpose  for  this  discussion  board  was  to  provide  a support  mechanism  for  facilitators  to  ask  for, 
and  provide  advice  to  other  facilitators.  A total  of  287  individuals  were  accepted  to  facilitate  in  FOR- PD, 

70  (25%  of  those  accepted)  of  which  were  facilitating  at  least  one  section  of  FOR-PD  as  of  September 
2003.  The  average  years  of  teaching  experience  for  facilitators  was  nearly  18  years  with  one-half  boasting 
at  least  17  years  of  experience.  Approximately  56%  of  facilitators  had  taken  an  online  course,  however 
only  12%  had  previous  experience  in  facilitating  online. 

Facilitator  Focus  Group  Report 

A facilitator  focus  group  was  conducted  in  May  2003  at  UCF.  Twenty-seven  facilitators  attended 
the  focus  group  meeting.  The  focus  group  report  provided  a broad  picture  of  the  FOR-PD  system  from 
facilitator’s  perspective.  The  general  tone  of  responses  from  the  focus  group  was  very  positive.  Facilitators 
appreciated  the  current  research-based  content,  the  database  of  materials  and  resources  developed  as  part  of 
FOR-PD.  However,  issues  and  concerns  were  raised  in  the  focus  group,  too.  For  example,  some 
participants  had  copied  from  each  other  and  had  submitted  work  that  had  been  directly  copied  from  the 
Internet.  In  addition,  facilitators  listed  the  WebCT  grade  book  as  their  No.  1 technology  problem.  Finally, 
the  report  gave  eight  recommendations  for  full-time  FOR-PD  staff  to  consider  and  stated  that  many  of 
those  (concerns  and  issues)  had  been  addressed  through  the  work  of  the  editing  team  several  weeks  after 
the  focus  group  meeting  and  the  current  course  revision  underway  (before  the  beginning  of  phase  2)” 

Methods 

The  site  and  major  data  source  for  the  current  qualitative  evaluation  was  the  facilitators’ 
discussion  board  postings  from  fall  September  2003.  There  were  161  postings  as  of  December  31,  2003, 
including  topics  such  as:  meeting  peer  facilitators  and  introducing  their  backgrounds,  making  comments, 
extending  greetings  to  each  other  for  the  new  semester  2003  fall,  offering  help,  hints  and  advice,  sharing 
success  stories  of  FOR-PD  participant  learning,  providing  suggestions  the  FOR-PD  facilitator’s  electronic 
newsletters,  providing  general  ideas  and  suggestions,  posting  messages  related  or  unrelated  to  FOR-PD 
course,  and  discussing  Lesson  1 to  Lesson  14.  The  data  used  for  the  evaluation  consisted  of  120  messages 
posted  by  the  facilitators  out  of  the  total  of  161  on  the  discussion  board  since  the  beginning  of  phase  2.  As 
the  major  source  of  data  for  the  present  evaluation,  facilitator’s  discussion  board  met  some  requirements  of 
the  RFP  and  a few  focuses  indicated  in  the  UCF  proposal  for  FOR-PD. 

Robert  Yin's  book.  Case  study  research:  design  and  methods  (2nd  ed.)  was  used  for  the  design  for  this 
qualitative  evaluation  and  one  of  its  dominant  mode  of  case  study  analysis,  the  combination  of  ‘pattern  - 
matching’  and  ‘time -series  analysis’  was  applied  to  analyze  and  explain  facilitators’  perceptions  and 
experiences  of  FOR-PD.  Moreover,  with  the  qualitative  software  Nvivo  Revision  [1.3],  automatic  coding  of 
the  data  was  used  in  addition  to  hand  coding  and  various  codes  were  developed  (see  details  in  Appendix: 
Data  Analysis). 


Findings 

From  the  Facilitators’  discussion  board  postings,  two  major  categories  emerged,  messages 
conveying  facilitator’s  comments  and  messages  conveying  facilitator’s  activities.  Meanwhile,  the 
frequency  of  postings  on  the  discussion  board  varied  largely  from  month  to  month  at  different  data  points: 
September,  October,  November  and  December.  As  Table  1 shows,  out  of  the  total  161  posts  including 
coordinator  or  instructor’s  messages,  there  are  95  more  in  the  first  half  of  the  semester  (September  and 
October)  than  the  later  half  (November  and  December). 

Table  1 Monthly  Posts  in  Facilitator’s  Discussion  Board  Phase  2 


Month 

Number  of  Posts 

September 

89 

October 

39 

November 

20 

December 

13 
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Facilitators’  Comments 

Operationally,  ‘encouragement’  included  expressions  like  ‘look  forward  to  new  session’,  ‘like  the 
changes  since  summer’,  ‘learn  a lot’  from  FOR-PD,  and  etc,  while  ‘criticism’  connoted  ‘frustrating  about 
participants  who  dropped  out’,  ‘unavailable  assistance’  and  etc.  In  terms  of  comments  (see  Table  2),  the 
first  half  of  the  semester  had  a contrastingly  larger  amount  of  ‘encouragement’  from  facilitators  than  the 
later  half  of  the  semester,  with  a ratio  of  57  to  three.  Likewise,  15  negative  messages  appeared  in  the  first 
half  while  none  in  the  later  half.  In  general,  there  were  a lot  more  ‘encouragement’  than  ‘criticism’  during 
the  whole  section. 

‘Encouragement’  messages  were  divided  into  six  categories  (Table  3).  Specifically,  a lot  of 
facilitators  expressed  cheerfulness,  looked  forward  to  new  session  and  liked  the  changes  of  layout  and 
content  since  summer. 


Table  2 Monthly  Posts  of  Comments  in  Facilitators’  Discussion  Board  Phase  2 


Comments 

encouragement 

criticism 

September 

50 

8 

October 

7 

7 

November 

2 

0 

December 

1 

0 

Table  3 Facilitators’  Comments  in  the  Discussion  Board  in  Phase  II 


Facilitators’  Comments 

Encouragement 

Sept. 

Oct. 

Nov. 

Dec. 

look  forward  to  new  session  and  like  the  changes,  including 
the  scoring  rubric 

X 

X 

X 

enjoy  being  a facilitator,  and  show  pride  in  completion  rate 

X 

X 

like  FOR-PD  course 

X 

appreciate  facilitator  discussion  boards 

X 

X 

Identify  with  participants'  encouragement 

X 

share  influence  of  for-pd  course  in  both  schools  and  families, 
something  beyond  participants’  learning 

X 

X 

X 

Total  Monthly  No. 

50 

7 

2 

1 

Criticism 

Sept. 

Oct. 

Nov. 

Dec. 

frustration  over  participants’  low  completion  rate 

X 

complaints/confusion  about  technical  problems,  some  caused 
by  the  changes  in  the  new  session 

X 

X 

empathy  with  participants’  frustration  over  assistance  from 
FOR-PD  project  staff,  such  as  help  desk 

X 

X 

Total  Monthly  No. 

8 

7 

0 

0 

Following  are  quotes  identified  and  extracted  from  the  discussion  board  for  ‘encouragement’  messages 
from  facilitators. 

Iam  excited  to  see  the  final  version  of  the  changes.  I know  the  previews  were  impressive.  Iam 
looking  forward  to  the  start  of  the  new  sections...  1 look  forward  to  my  next  chance  to  facilitate. 
Until  that  time  I plan  on  regular  visits  to  the  Discussion  area  to  see  what  exciting  things  continue 
to  transpire  in  the  wonderful  land  of  FOR-PD.  ” 

“I  look  forward  to  the  new  format.  So  far,  it  looks  great  and  seems  so  much  more  user  friendly.  Of 
course,  I didn 't  realize  that  the  old  course  could  be  improved  upon  that  much!  ” 

“The  summer  course  design  was  SO  much  easier  to  manage.  I just  wish  that  more  teachers  would 
take  advantage  of  this  wonderful  opportunity.  ” 

“The  changes  that  were  made  for  the  fall  are  awesome!  Thanks  for  all  of  the  hard  work  FOR-PD 
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staff... you  guys  are  great.  ” 

Other  facilitators  enjoyed  being  a facilitator  or  liked  the  FOR-PD  course  itself.  Following  are 
examples. 

“Catherine.  This  is  not  a difficult  course  to  facilitate.  I did  it  for  the  first  time  this  summer  and  I 
loved  it.  ” 

“I  loved  facilitating  this  summer.  This  is  a terrific  course.  I've  learned  a lot”. 

“I  took  the  FOR-PD  course  as  a student  last  fall  and  was  impressed...  I found  that  the  FOR-PD 
course  offered  that  and  a window  to  the  ever-changing  legislative  directives.  The  more  I learn  the 
more  1 can  share  with  my  students,  teachers  and  parents.  ” 

“We  had  a third  grade  unit  open  up  last  week  and  I will  be  moving  to  the  3rd  grade  unit  on  the 
18th.  I am  very  excited  and  plan  on  using  much  of  what  I have  learned  here  in  my  class”. 

“It  was  a great  experience  for  me  to  travel  to  different  places  around  the  state.  It's  shame  that  we 
really  don't  get  to  interact  with  more  educators  in  different  districts.  That's  one  of  the  great 
benefits  of  this  course.  ” 

“The  FOR-PD  class  has  been  a big  part  of  my  life  since  March.  I carried  my  laptop  cdl  summer 
long  as  I traveled  through  the  state  with  my  daughter's  softball  team.  I was  not  given  a new 
section  to  facilitate  and  I will  miss  the  interaction,  the  lesson  discussions,  and  the  daily 
information  bank  on  literacy!” 

A few  more  appreciated  the  role  of  the  facilitator  discussion  board. 

“Well  I'm  thankful  for  the  facilitator  discussion  boards  because  I think  this  will  be  my  lifeline. 

This  is  my  first  time  facilitating  a course  & honestly  I'm  a little  nervous.  ” 

“This  site  will  save  you  when  you  need  really  need  it!” 

“I  am  looking  forward  to  learn  ingfrom  all  the  experts  who  have  already  facilitated  this  course 
and  I know  I will  get  a lot  of  use  out  of  this  discussion  board.  ” 

A few  facilitators  identified  with  participants’  satisfaction  with  the  course. 

“I  look  forward  to  the  new  look  of  the  course  and  hope  to  hear  many  more  wonderful  things  about 
this  course.  I had  a participant  show  how  much  she  enjoyed  this  course  by  making  it  part  of  her  plan 
(lesson  14).  She  is  going  to  be  working  with  her  administration  to  try  to  convince  everyone  that  they 
need  to  take  this  course.  That  was  a big  “Wow”  to  me”. 

“I  truly  enjoy  the  notes  I've  received  from  course  participants  sharing  how  they've  used  strategies 
from  the  FOR-PD  course  in  their  classrooms.  It's  exciting  to  be  a part  of  the  process  of  having 
EVERY  teacher  become  a reading  teacher 1” 

“We  have  had  a great  response  from  our  teachers...  ” 

Moreover,  some  extremely  positive  themes  that  occurred  in  the  discussion  board,  including  the 
influence  of  FOR-PD  course  in  schools  and  families,  the  impact  beyond  participants’  learning  success. 

“Our  SAC  wrote  the  course  into  our  School  Improvement  Plan  this  year,  so  all  of  our  instructors 
who  teach  high  school  students  will  be  taking  the  course.  Several  of  our  instructors  who  teach 
only  adult  students  opted  to  take  it  also!  ” 

“What  a great  idea!  I am  SAC  coordinator  at  my  school,  along  with  all  my  other  jobs. ..I  think 
that  writing  the  course  into  the  SIP  is  a great  idea.  I'll  have  to  look  into  it.  ” 

“And  YES,  we  are  currently  applying  FOR-PD  to  Competency  #2  in  our  district  Reading 
Endorsement  Plan.  1 am  very  happy  to  be  involved  in  this  professional  development  activity.  I 
believe  that  we  will  have  a better  completion  rate  this  fall  than  in  the  past.  ” 

“ This  was  shared  by  one  of  my  course  participants....  ‘I  also  want  to  tell  you  that  when  I was  doing 
the  ABC  brainstorming  page  for  lesson  4 (I  think),  my  5 year  old  daughter  was  watching  and 
asking  what  I was  doing.  She  asked  me  to  print  a page  for  her,  which  I did,  and  she  did  all  by 
herself.  I am  enclosing  it  in  my  literacy  log.  I think  you  'll  get  a kick  out  of  it.  ’ Just  shows  that  we 
never  know  what  effect  this  course  might  have  on  others!!  :-)  ” 

Table  3 also  shows  through  the  whole  course,  there  were  15  messages  that  facilitators  provided 
‘criticism,’  all  of  which  appeared  in  the  first  half  of  semester.  Some  conveyed  facilitator’s  confusion  and 
frustration  over  technical  problems,  including  those  that  might  be  caused  by  the  change  since  summer,  or 
facilitator’s  uneasiness  about  participant’s  low  completion  rate  “It's  been  frustrating  because  of  participants 
who  didn't  complete.”  Moreover,  a few  facilitators  expressed  they  could  not  reach  administrative  and 
technical  support,  such  as  the  Help  Desk.  Following  are  a series  of  criticism  at  the  end  of  October. 

“I  just  wanted  to  pass  two  things  along....  1.  Severed  of  my  participants  have  been  complaining 
about  the  help  desk  ....that  they  have  not  been  returning  phone  calls  nor  have  they  been  able  to 


920 


reach  a person  during  the  hours  posted.  2.  It  seems  that  there  has  been  many  more  technical 
problems  than  any  other  section  ( this  is  my  third  section),  is  there  a reason  for  that?  Participants 
cannot  get  to  the  site,  they  cannot  get  a quiz  or  it  does  not  post  until  several  tries,  they  cannot  get 
to  links.  ” 

“This  is  my  fourth  section  of  the  course  and  my  participants  have  had  more  technical  issues  this 
time  than  before.  Initially,  I thought  it  was  growing  pains 

with  our  district  network,  but  we  have  supposedly  solved  those  problems,  yet  participants  are  still 
complaining  about  various  technical  problems.  Any  insight?” 

“1  have  to  agree,  this  is  the  third  time  I have  been  a facilitator  and  there  have  been  many 
technical  difficulties...  could  this  be  because  of  the  newly  designed  site? 

“ That  is  actually  what  I was  thinking,  because  they  resigned  the  site  right  after  the  end  of  the  last 
section  ....  before  that  my  participants  did  not  have  any  of  these  problems  ...  that  is  a good  point!” 
Facilitator’s  Activities  and  Contributions 

One  great  use  of  the  discussion  forum  was  to  ask  for  help,  report  to  the  FOR-PD  program 
coordinator  about  problems  and  errors,  and  respond  or  give  suggestions  to  questions  raised  by  other  peer 
facilitators  (see  Table  4).  Facilitators’  posts  of  activities,  similar  to  their  comments,  were  posted  more 
frequently  in  the  first  half  of  the  semester  than  in  the  later  half,  with  a ratio  of  43:18. 


Table  4 Facilitator’s  Activities  in  the  Discussion  Board  in  Phase  2 


Facilitator’s  Activities 

Sept. 

Oct. 

Nov. 

Dec. 

ask  for  information;  seeking  help  including  how  they  could 
solve  problems  for  participants’ 

X 

X 

X 

X 

report  to  FOR-PD  UCF  program  coordinator  and  discuss 
with  peer  facilitators  about  technical,  administrative  and 
content  problems  and  errors  in  different  aspects,  including 
access  to  quizzes  and  participants’  course  pacing. 

X 

X 

X 

X 

respond/give  suggestions  to  questions  from  peer  facilitators 

X 

X 

X 

X 

Total  Monthly  No. 

27 

16 

11 

7 

Examples  of  comments  from  facilitators  asking  for  information  or  seeking  help,  including  how  they  could 
solve  problems  for  participants,  follows. 

“Could  you  please  send  me  more  brochures?  1 want  to  share  them  with  the  faculty  again.  Some 
people  have  shown  an  interest  in  taking  the  course.  Is  it  possible  for  someone  to  sign  up  now  for 
the  fall  course?” 

“I  know  this  information  is  in  my  manned,  but  I'm  at  school.  I need  a refresher  on  how  to  close  a 
session,  ” 

“ How  do  you  manually  go  back  in  and  change  the  score?  1 need  to  fix  a few  of  my  participants’ 
scores.  ” 

“ Since  this  is  my  first  time  facilitating  the  course,  I'm  not  sure  where  I should  post  general 
feedback  to  the  discussion.  Do  I create  a new  message  in  lesson  1 or 
somewhere  else?  Most  of  my  participants  have  completed  lesson  1.  I would  like  to  post  a 
generalization  about  the  discussion  & test.  If  I have  any  specific  comments  to  participants  I will 
send  it  directly  to  the  participants.  Thanks  for  your  help  with  this.  ” 

“I  have  severed  participants  who  are  getting  the  following  message  when  they  go  to  take 
quizzes. ..I  have  emailed  them  about  the  pop-  ups. ..is  there  any  other  reason  it  may  be  this  way?? 
Here  is  the  question  l received:  Susan,  this  past  weekend  I was  on  a different  computer. 
However  today  is  Monday  and  I am  back  on  the  computer  that  I have  been  using.  I still 
cannot  take  a quiz.  In  fact,  all  of  the  quizzes  are  marked  unavailable. 

I'd  like  to  ease  these  participants'  troubles.  ” 

An  example  of  a response  to  a question  raised  by  a peer  facilitator  (for  example,  a response  to  the  last 
question  in  the  above  paragraph)  was  phrased,  “sounds  crazy,  but  has  she  taken  the  survey?  1 had  a few 
contact  me  about  not  being  able  to  take  lesson  2 quiz,  but  they  hadn  't  taken  the  survey.  As  soon  as  they  did, 
all  was  okay.  ” 

Following  is  a series  of  posts  concerning  accessing  quizzes  which  provides  an  example  of 
reporting  correspondence  with  FOR-PD  UCF  program  coordinator  and  discussion  with  peer  facilitators 
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about  problems  and  errors. 

“I  have  a participant  that  can  not  access  quiz  3.  I have  called  the  help  desk , she  has  called  the 
help  desk,  I have  tried  her  in  every  way  that  I can.  She  has  submitted  lesson  2 's  quizzes  and  they 
have  been  graded.  She  still  cannot  access  the  quizzes.  Now  she  is  considering  dropping  from 
FOR-PD.  What  can  / do  ? We  can 't  figure  out  why  she  can 't  access  the  quizzes.  The  Help  Desk  sees 
no  reason  why  she  couldn 't  access  the  quizzes.  I am  at  a loss!!!!  ” 

“Has  she  done  all  the  quizzes  prior  to  Lesson  3 and  the  pre-course  survey  and  have  grades  been 
posted  for  all?  I do  know  that  it  will  not  let  you  skip  a quiz.  Has  she  tried  to  use  another 
computer,  some  of  my  participants  are  having  difficulties  with  their  school  computers” . 

“I  had  a participant  who  couldn't  do  the  quizzes  because  the  pop-ups  were  turned  off  on  her 
computer.  That's  yet  another  thing  to  check....  ” 


Evaluation  Summary 

The  investigation  of  facilitator’s  discussion  board  revealed  that  the  overall  pattern  of  facilitators’ 
use  of  the  forum  matched  the  expectations  and  requirements  as  indicated  in  the  Request  for  Proposal  (RFP) 
of  Florida  Department  of  Education  and  the  UCF  proposal  for  FOR-PD  system.  Facilitators’  activities  and 
contributions  ranged  from  giving  technical  support,  monitoring  of  content  including  assignments,  rubrics 
and  course  pacing,  to  facilitating  interaction  between  FOR-PD  Help  Desk  and  participants.  Moreover, 
facilitators  generally  provided  encouragement  for  the  revised  FOR-PD  course  (phase  II),  including  its  new 
layout  such  as  the  facilitators’  forum  and  its  new  content  such  as  the  rubrics.  However,  some  facilitators 
still  had  certain  confusion  and  frustration  over  the  change  of  the  course,  particularly  about  the  access  to 
quizzes  and  other  technical  problems,  and  about  the  availability  of  Help  Desk  in  terms  of  administrative 
support  during  the  first  half  of  the  semester  (September  and  October  2003). 

Compared  with  the  facilitator  focus  group  report  done  May  2003,  which  provided  a broad  picture 
of  FOR-PD  system  from  facilitators’  perspective  as  the  preliminary  data  for  this  evaluation,  the  discussion 
boards  revealed  to  the  current  researchers  how  facilitators  played  their  roles  and  helped  participants 
enhance  learning  in  a much  greater  depth.  In  addition,  the  ‘time  -series’  analysis  of  the  discussion  board  has 
shown  that  FOR-PD  was  getting  better  over  time  although  confusion  and  problems  still  existed  in  phase  II. 
To  be  specific,  good  themes  had  been  maintained,  as  seen  in  both  the  focus  group  data  in  phase  I and  the 
data  from  the  discussion  board  in  phase  II.  For  example,  the  general  tone  of  responses  from  both  data 
sources  indicated  that  facilitators  appreciated  the  current  research-based  content,  and  the  database  of 
materials  and  resources  developed  as  part  of  FOR-PD.  In  addition,  facilitators  did  not  mention  the  WebCT 
grade  book  as  their  technology  program  any  more  in  phase  II.  Neither  did  the  facilitators  suggest  that 
interaction  among  participants  were  inhibited  in  this  section  as  in  the  focus  group  report.  To  sum  up  with 
one  post  in  the  facilitator’s  discussion  board,  “I  am  surprised  there  is  so  little  activity  on  the  discussion 
board  this  section.  I guess  so  many  of  us  are  ‘old-timers’  that  most  of  the  questions  have  been  answered.” 

Moreover,  a trend  was  also  found  at  different  points  of  phase  II,  that  is,  more  activities  and 
comments  of  facilitators  appeared  on  the  discussion  board  in  the  first  half  of  phase  II  than  the  later  half  in 
November  and  December,  which  implied  that  facilitators  had  helped  participants  solve  their  learning 
problems  in  a timely  fashion. 


Recommendations 

The  findings  from  the  facilitator’s  discussion  board  and  the  comparison  made  with  the  previous 
facilitator  focus  group  also  showed  several  ongoing  problems  in  FOR-PD  program.  Following  are  a few 
recommendations  to  address  them  and  a couple  of  suggestions  for  the  final  qualitative  evaluation  of  FOR- 
PD  program  planned  for  May  2004. 

The  Help  Desk  currently  has  a goal  of  addressing  and  resolving  technical  problems  within  a 24- 
hour  period.  Careful  attention  to  continuing  quick  response  should  be  monitored  to  ensure  efficient  and 
effective  support  in  response  to  problems  encountered  by  FOR-PD  facilitators  and  participants.2 

Facilitators  should  not  only  be  updated  and  familiarized  before  changes  are  made  in  the  FOR-PD 
system  to  ensure  they  are  comfortable  with  and  understand  how  the  changes  will  impact  the  course  and  can 
thereby  be  more  effective  in  assisting  participants,  but  they  should  also  be  reminded  about  the  changes — 
for  example,  the  changes  of  access  to  quizzes  caused  some  confusion  with  facilitators.3 

While  FOR-PD  has  been  overall  effective  in  using  the  online  system,  various  technical  problems 
have  been  frustrating  to  some  participants  and  facilitators,  which  may  or  may  not  have  been  problems 
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within  FOR-PD  itself.  It  is  suggested  that  attention  to  improving  the  technical  aspects  of  FOR-PD  and 
researching  and  implementing  ways  to  make  the  technology  more  user-friendly  should  be  continued. 

To  understand  the  factors  that  have  an  impact  on  FOR-PD  better,  additional  qualitative  analyses 
should  be  conducted  including  telephone/on-line  interviews  and/or  focus  groups  with  five  key  audiences: 
school  districts,  participants,  facilitators,  Florida  Department  of  Education  staff,  and  FOR-PD  content 
contributors  or  course  designers/instructors. 
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Appendix:  Data  Analysis 

Robert  Yin's  book.  Case  study  research:  design  and  methods  (2nd  ed.)  was  used  for  the  design  for 
this  qualitative  evaluation  and  one  of  its  dominant  mode  of  case  study  analysis,  the  combination  of  ‘pattern  - 
matching’  and  ‘time  -series  analysis’  was  applied  to  analyze  and  explain  facilitators’  perceptions  and 
experiences  of  FOR-PD  phase  II.  Pattern  matching  “compare  an  empirically  based  pattern  with  a predicted 
one  (or  with  several  alternative  predictions)  for  dependent  variables’’  (Yin,  1994).  In  this  case,  the  outcome 
of  the  use  of  facilitators’  discussion  board  was  matched  with  the  requirements  of  the  RFP  of  Florida  DOE 
and  the  focuses  of  the  UCF  proposal  for  FOR-PD  program.  On  the  other  hand,  time  -series  analysis  or  “the 
match  between  a trend  of  data  points  compared  with  a theoretically  significant  trend  specified  before  the 
onset  of  the  investigation’’  (Yin,  1994)  was  applied,  too.  In  this  paper,  data  points  were  set  in  each  month: 
September,  October,  November  and  December;  time  -series  analysis  also  included  a comparison  of  the 
outcome  of  the  investigation  into  the  facilitator’s  discussion  board  with  the  facilitator’s  focus  group  report 
done  May  2003. 

Moreover,  with  the  qualitative  software  Nvivo  Revision  [1.3],  automatic  coding  of  the  data  was 
used  in  addition  to  hand  coding.  Firstly,  all  161  messages  were  copied  and  imported  to  the  Nvivo  system, 
forming  six  major  documents  representing  different  genre  of  the  data:  Facfocus  (the  facilitator  focus  group 
report),  Facilitator  E-community,  Problems,  Suggestions  and  Comments,  Success  Stories,  and 
Miscellaneous.  Next,  dozens  of  codes  were  developed  following  the  basic  guideline  of  the  role,  time,  and 
content  of  the  message,  forming  three  dimensions  in  the  coding  process.  By  reviewing  the  data  and  the 
codes,  or  namely,  ‘nodes’  as  termed  in  the  Nvivo  system,  the  codes  indicating  the  role  of  facilitators  and 
those  explaining  the  content  of  the  message  were  combined  into  two  big  categories:  facilitators’ 

‘comments’  and  ‘activities’.  Later,  using  cross-search  of  the  Nvivo  system,  some  within  the  two 
dimensions  of  ‘time’  and  ‘comments’,  and  others  within  the  two  dimensions  of  ‘time’  and  ‘activities’, 
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several  ‘trees’  of  the  ‘nodes’  as  termed  in  the  Nvivo  were  formed.  Finally  those  ‘trees’  of  codes  became  the 
main  themes  of  the  data  and  were  quoted  as  the  findings  of  the  paper  under  the  two  big  categories: 
‘facilitator’s  comments’  and  ‘facilitator’s  activities  and  contributions’. 
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